NetMassDownloader in VB.NET

Welcome to the .NET Mass Downloader project. While it's great that Microsoft has released the .NET Reference Source Code, you can only get it one file at a time while you're debugging. If you'd like to batch download it for reading or to populate the cache, you'd have to write a program that instantiated and called each method in the Framework Class Library. Fortunately, .NET Mass Downloader comes to the rescue!

The tool which enables offline debugging in VS2008 , VS2008 Express Edition ,VS2005 And Codegear Rad Studio. Only thing you need to use this tools is having .Net Framework 3.5 RTM Installed. With it , you can have whole the source code without any Visual Studio Product Installed.

Background

As you all know the .Net Framework 3.5 Source Code has been made public via Visual Studio 2008 with the install of a small addin a few weeks ago.The actual short coming of the tool was, that it didn't support the download of the whole framework code at once, nor it has been downloading each source file for a class one by one. Also because of each time downloads it was not possible to use this feature offline , so no offline debugging was possible.

It was also not possible to use this nice feature in VS2005 and VS2008 Express Editions. Our tool enables you to use whole this missing features.

Some small note from John Robbins Blog About Our Project:

Kerem Kusmezer contacted me and said that he'd been hacking around with Fiddler and figured out how to download the PDBs and code from Microsoft. He asked if I I'd be interested in helping him develop the utility and I don't think I ever wrote a response faster in my life! Kerem had already started a discussion with Shawn Burke about the legality of producing a source downloader and once we got back a very encouraging "go for it" we got to work. All the credit for the tool belongs to Kerem Kusmezer. He did all the really hard work of figuring out the HTTP downloads ,the PE file processing, and PDB Stream Extraction.

I did the console driver, a lot of testing, and sat around amazed at how much work Kerem could accomplish in very short periods of time. I really appreciate that Kerem let me work on the project with him and being his code monkey.

Prerequisites:

When you first run Net Mass Downloader, you will be prompted with the current EULA for accessing the source code. If you don't agree with the Microsoft EULA, clicking the Decline button will not download the source code.

You must install the RTM Version of the .NET Framework 3.5 to access the source codes even if you are using VS2005.

Using the code

The tools has been implemented as a commandline application , which shows the following help screen after you execute it.

The only required arguments are -file or -directory, both of which can be specified as many times as you'd like. When you specify a directory, only the .DLL and .EXE files from that directory will be processed. If you wanted to download all the source code from binaries in the .NET 2.0 32-bit and 64-bit directories, the command line you'd pass is: -d C:\Windows\Microsoft.NET\Framework\v2.0.50727 -d C:\Windows\Microsoft.NET\Framework64\v2.0.50727.

The main purpose of Net Mass Downloader is to populate the source code download cache for debugging, the default download location is the cache you specified to Visual Studio 2008. We put in the -vsver switch to account for future Visual Studio versions so Mass Downloader could work with future CTPs and versions.While it's great to see the .NET Reference Source Code in Visual Studio 2008, there are a lot of developers out there who can't upgrade yet.

VS2005 Setup For NetMassDownloader

Initially, we went down the path of seeing if we could patch the PDB files so Visual Studio 2005 users could have access to the .NET Reference Source Code. After a lot of hassle, we realized that there was a much easier way. If you specify the -output switch, we will put the PDB and the source code into that directory using the build paths instead of the source server paths. Thus, if your command line to NetMassDownloader.exe is -d C:\Windows\Microsoft.NET\Framework\v2.0.50727 -output c:\ReferenceSource, the PDB files are put in C:\ReferenceSource as well as the source files.

To use the .NET Reference Source with Visual Studio 2005, you would need to do the following three additional steps. The first is in the Options dialog for Visual Studio, Debugging, Symbols property page. In the "Symbol file (.pdb) locations" list box, you would add "c:\ReferenceSource" like the following:

The second setting is in Options dialog, Debugging, General property page, uncheck Require source files to exactly match the original version.

The final step you'll have to do in each solution where you want to access the downloaded .NET Reference Source Code, you'll go into the solution property pages, Common Properties, Debugging Source Files, and add C:\ReferenceSources to the top of the "Directories containing source code" list box as follows:

Visual Studio 2005 will automatically do the right thing and replace the first directory of every source file it finds in a PDB file with c:\ReferenceSource. With the automation model in Visual Studio 2005, I'm sure you could wipe up a macro to automatically set that path in the solution.

VS2008 Express Edition Setup

Initially, we went down the path of seeing if we could patch the PDB files so Visual Studio 2005 users could have access to the .NET Reference Source Code. After a lot of hassle, we realized that there was a much easier way. If you specify the -output switch, we will put the PDB and the source code into that directory using the build paths instead of the source server paths. Thus, if your command line to NetMassDownloader.exe is -d C:\Windows\Microsoft.NET\Framework\v2.0.50727 -output c:\ReferenceSource, the PDB files are put in C:\ReferenceSource as well as the source files.

To use the .NET Reference Source with Visual Studio 2008 Express Edition, you would need to do the following three additional steps. The first is in the Options dialog for Visual Studio, Debugging, Symbols property page. In the "Symbol file (.pdb) locations" list box, you would add "c:\ReferenceSource" like the following:

First be sure that "Require source files to exactly match the original version" is not checked in VS2008 Options -> Debugging -> General Dialog.

After that check the checkbox Show all settings which is at the left down of the screen. If show all settings has been checked, the screen will look like the following:

There please goto the Symbols location , add the target path that you give out in the -output directory there. For example we assume you used the following commandline arguments to execute NetMassDownloader
netmassdownloader.exe -d c:\winnt\microsoft.net\framework\v2.0.50727 -output c:\cachetest -v

You will see a similar window like this:

Just write the output directory c:\cachetest as above to the symbols dialog. Only one step is left after that, you can enjoy debugging the .net framework under VS2008 Express Edition. Just open a new solution in VS2008 Express Edition, goto the Solution Properties Dialog, add the above mentioned directory to it like below:

Now you can debug with F11 into the sourcecode of .net framework offline from your VS2008 Express Edition.

Additional Notes

When you first run Net Mass Downloader, you will be prompted with the current EULA for accessing the source code. If you don't agree with the Microsoft EULA, clicking the Decline button will not download the source code. While you might be tempted to remove the EULA check, I would strongly recommend that you not do so as Microsoft has said that they may change the EULA at any time. The last thing you want to do is get caught with a license change that could impact your life.

One group we need to thank is the Developer Division at Microsoft. First they released the .NET Reference Source Code, and second for allowing a couple of developers to have some fun and provide a utility for the community. As always let us know on CodePlex if you encounter any bugs or have feature requests.

Points of Interest

Things i have learn during the course of the implementation are as following:

Decomposition and parsing of a pdb file. Actually I ported a pdb parser to c# for this reason.

Decomposition and parsing of a pe file, finding the required debug information sections in it.

A srcsrv file parser , which also parses srcsrv macros included in the srcsrv files.

A customer webclient class, which helps with eula downloads.

Project Structure

The solution consist of 2 projects : DownloadLibrary and NetMassDownloader Console Application.

When we break down each component of the DownloadLibrary Class Library, they are the following:

DownloadLibrary.PEParsing.PEFile The class which is responsible for PE Header and Debug Directory Parsing and detection , also responsible for PDB Download and Url Creation. Uses the IMAGE_DOS_HEADER, PESectionHeader,CV_INFO_PDB70, _IMAGE_DEBUG_DIRECTORY, IMAGE_DATA_DIRECTORY, IMAGE_FILE_HEADER, IMAGE_NT_HEADERS32, IMAGE_NT_HEADER64 , IMAGE_OPTIONAL_HEADER32, IMAGE_OPTIONAL_HEADER64 Structures. Age, Guid , Version of the target PDB File is found out using this structures.
DownloadLibrary.Classes.PdbParser The class is responsible for PDB Parsing. It is actually part of a different project also written by me. PdbParser For C#, which will be released in a few weeks. It supports parsing and stream decomposition of a PDB70 Version PDB File. It extracts whole streams, parses the PDB Header etc. This is required to parse out the SRCSRV file , which holds the url locations of each source code file.
DownloadLibrary.Classes.PdbExtractor This class is responsible for creating SrvSrcFile from the provided PdbFile, also responsible for downloading each source file from the Microsoft Reference Server and putting them in correct directories. It uses the PDBWebclient, PdbParser , ScrSrvFile classes for that purpose.
DownloadLibrary.Classes.SrcSrvDownloadableFile,SrcSrvFile These classes are responsible for SrcSrv files parsing , target url generation and target path generation.
DownloadLibrary.Classes.Eula.PDBWebclient This is a class which is derived from System.Net.WebClient which is responsible for downloading the contents from Microsoft Reference Server, Handling and Storing Of Eula Response And Eula Detection.

History

v1.1 A Bugfix Release Has Been Released Which Solves A Bug With Some Proxies Not Returning The Correct Headers.

Download NetMassDownloader From Here For Free(http://www.codeplex.com/netmassdownloader)

You can download also the source code from the project page.