Web Scraping In C# Using ScraperAPI

static async Task GetDataFromWebPage() {
  try {
    Console.WriteLine("### Started Getting Data.");

    string apiKey = "**Add Your API Key Here**";
    HttpClient scraperApiHttpClient = ScraperApiClient.GetProxyHttpClient(apiKey);
    scraperApiHttpClient.BaseAddress = new Uri("https://coinmarketcap.com");

    var response = await scraperApiHttpClient.GetAsync("/");
    if (response.StatusCode == HttpStatusCode.OK) {
      var htmlData = await response.Content.ReadAsStringAsync();
      ParseHtml(htmlData);
    }
  } catch (Exception ex) {
    Console.WriteLine("GetDataFromWebPage Failed: {0}", ex.Message);
  }
}

Replace your ScraperAPI key in the “apiKey” variable.

GetProxyHttpClient() is used to create a HTTP client with the scraperapi.com proxy.

GetAsync() will fetch the data from the website and store in local variable.

Step 4 - ParseHtml() method

Once get HTML data from Webpage, parsing it using the HTMLdocument method. It comes from HtmlAgilityPack.

Next step, load HTML data and get the ‘tbody’ HTML tag from it. The tbody tag contains the rows of Cryptocurrency data.

To get more data, use selectSignleNode method. It will return the first HtmlNode that matches the XPath query, it will return a null reference if the matching node is not found. SelectNodes is a collection of Html Nodes.

static void ParseHtml(string htmlData) {
  try {
    Console.WriteLine("### Started Parsing HTML.");
    var coinData = new Dictionary < string,
      string > ();
    HtmlDocument htmlDoc = new HtmlDocument();
    htmlDoc.LoadHtml(htmlData);

    var theHTML = htmlDoc.DocumentNode.SelectSingleNode("html//body");
    var cmcTableBody = theHTML.SelectSingleNode("//tbody");
    var cmcTableRows = cmcTableBody.SelectNodes("tr");
    if (cmcTableRows != null) {
      foreach(HtmlNode row in cmcTableRows) {
        var cmcTableColumns = row.SelectNodes("td");
        string name = cmcTableColumns[2].InnerText;
        string price = cmcTableColumns[3].InnerText;
        coinData.Add(name, price);
      }
    }
    WriteDataToCSV(coinData);
  } catch (Exception ex) {
    Console.WriteLine("ParseHtml Failed: {0}", ex.Message);
  }
}

Step 5 - WriteDataToCSV() method

In this example, we have taken the currency name and its price from the scraped data, and store it in a CSV file.

static void WriteDataToCSV(Dictionary < string, string > cryptoCurrencyData) {
  try {
    var csvBuilder = new StringBuilder();

    csvBuilder.AppendLine("Name,Price");
    foreach(var item in cryptoCurrencyData) {
      csvBuilder.AppendLine(string.Format("{0},\"{1}\"", item.Key, item.Value));
    }
    File.WriteAllText("C:\\Cryptocurrency-Prices.csv", csvBuilder.ToString());

    Console.WriteLine("### Completed Writting Data To CSV File.");
  } catch (Exception ex) {
    Console.WriteLine("WriteDataToCSV Failed: {0}", ex.Message);
  }
}

Step 6 - Main() method

Replace content of Main method with below code:

static async Task Main(string[] args) {
  await GetDataFromWebPage();
}

Output

Go to the path where you have created the CSV file and open that file. Here our CSV file path is “C:\Cryptocurrency-Prices.csv”.

We can see CSV file as output that contains two columns, 1) Currency Name and 2) Price. We can get required column and data based on our need.

Conclusion

In this article, with the help of ScraperAPI and HtmlAgility Nuget Packages, we can scrap the data from site, filter the require data and dump into CSV file.