Web Scraping In C#

1. Create a Console Application in C#.
2. Go the Solution Explorer. Right click the References. Click Manage Nuget Packages and then browse for “HtmlAgilityPack” and install it.

Just a little bit more about HtmlAgilityPack. It is a wrapper in C# that provides us the ability to query the Document Object Model and extract any sort of data that we want to. We are going to see this in action further.

After the installation of your package, come back to the “Program.cs” file and follow the code below to get our web Scraper running.

Create an instance of “HtmlWeb” which will load the HTML of the given URL using HTTP.



Now, I believe that you have noted down the class of the anchor tag as discussed above. So, we are going to use that class in our code. Now, write the following code.



Note that we have used two “//” and then the name of the element that we have identified and also the name of the class that we have noted down from the Document Object Model and converted that to the List, we can also take advantage of the LINQ using HtmlAgilityPack, so it depends upon your needs of what you want to do.

Now, in the final step, simply loop through the list and call the “InnerText” property of each item in the list.



Run it and you will get all the Header Names we have identified on our Web page. See the result below.



And there, you can see that we have successfully created a web Scraper in C# that takes out the data from the Yellow Pages depending upon our scenario.

What more?

Now, in a similar fashion let’s say you want to get the results that are on the next page of this website. So, always keep an eye on continuously changing the URL of the website. You will get a clue of what you really need. Take a look. For example, you want the results from the second page, so you will load the following URL in “HtmlWeb.Load()” method and will follow the same principle. See the page=2, that will take out the results from the second page of the search results.



Similarly, you can use it in desktop applications where you will just enter the name of the city and the search terms and will get the result. Just replace “software” with {0} and pass the value using string.format method to the URL. Then, send the request and you will get all the related results according to the input.