Generate C# Corner Statistics Data Based On Available Information On Website

Introduction

In this series of articles, we will see the complete process of preparing the C# Corner Statistics Dashboard. This is the first part of the series. Below is the flow of the series - 

1. How to Generate or prepare C# Corner Statistics data based on available information on the community website. The data is generated in JSON Format.
2. Use this generated C# Corner Statistics data and prepare charts based on that.
3. Using reusable Handlebar.js template and methods to prepare the Analytics Dashboard.

In this article, we will learn how to generate C# Corner statistics data using web scraping with the help of Cheerio.js. Most of you may have these questions in mind,

• Why should we prepare or generate C# Corner Statistics?
• Does C# Corner have any public API for developers to use site statistics?

I had these questions in my mind and I reached out to the C# Corner Team as well but due to other priority work or different current goals, they don't have Web API or data which I or other developers like me wanted to prepare or show the charts based on the C# Corner statistics.

 

C# Corner is the largest tech community and millions of developers from all over the globe visit this website monthly for reading and sharing new technical news, articles, blogs, videos, forum questions etc. based on cutting-edge technologies and latest developments. Also, people can connect to each other so as to gain more knowledge and theirs. So, we can say that C# Corner is a "Social Network Tech Community for Developers".

 

Since there is no API available I have searched and explored the C# Corner website to grab the potential and useful statistics for getting started and then got other related information too which is enough for preparing a C# Corner Charts Dashboard.

 

Which data we will prepare from C# Corner site?

• Total Number of technology categories served by the website (Approx. 152 with diverse range of technologies till 05-Jan-2019)
• Each technology category's contribution count in Article, Video, News, Blogs etc.

Based on this prepared JSON data, we can create lots of charts and some statistics as well.

 

Prerequisites

 

The following tools and technologies I have used to prepare the C# Corner site data.

• Node.js (As a simple Back-end Server)
• Cheerio JS (Core Part- As a Web Scrapping Utility Library)
• Fetch API ( For Async HTTP Request to web Page from the server side) 

Let's come to an important part of our article.

 

Step 1

 

Request the C# Corner web page for fetching all the technology categories from the node.js app.

 

Step 2

 

Once you get a response from the request, make another Async request to fetch each technology category data and store it to JSON variables.

 

Step 3

 

Once all the Requests are completed, save the final result to a JSON file.  The above-mentioned three steps are explained in the below code snippet with a proper comment.

 

We have used ES6 async and await in our requirement to call recursive function for the web request.

 

App.js 

// Import Perquisite
var fs = require('fs');
var cheerio = require('cheerio');
var fetch = require("node-fetch");

// Attaching a new function  toShortFormat()  to any instance of Date() class
//dd-mm-yyyy Format
Date.prototype.toShortFormat = function() {

        var day = this.getDate();
        var month = this.getMonth();
        var year = this.getFullYear();

        return "" + day + "-" + (month + 1) + "-" + year;
    }

var csharpURL = "https://www.c-sharpcorner.com/technologies";
var categoryResult = [];

//Utility Function wrapper for async request Written with ES6
const getArticleCountByCategory = async(url, article) => {
    const response = await fetch(url);
    const responseHtml = await response.text();

    return {
        html: responseHtml,
        article: article
    };
};


// Step-1 Request C-SharpCorner Web Page
getArticleCountByCategory(csharpURL, {}).then((html) => {
// Response Received from Step 1
    const $ = cheerio.load(html.html);
    var article = {};
    var executedCount = 0;
// Find total Category Count
    var totalCategory = $(".LinkNormalGray").length;
// Fetch Category name and hyperlink for Step 2
    $(".LinkNormalGray").each((index, item) => {
// Recursive Call or async request in each loop
        if (true || index < 2) {
            // console.log($(item).text());
            article = {
                category: $(item).text(),
                url: $(item).attr("href")
            };
 // Step 2 Once Get Response from Step 1 , Start Requesting Each category data
            var categoryCount = getArticleCountByCategory($(item).attr("href"), article).then((data) => {
                // Increament counter for executed request
                executedCount++;
                const $$ = cheerio.load(data.html);
                // console.log(data.article);
// Prepare JSON Data based on resonse
                data.article.categoryData = [];
                var articleCount = $$('.count').each((index1, item1) => {
                    if (true || index1 < 2) {
// Push Category wise data to JSON Object
                        var categoryWithCount = $$(item1).parent().attr('title').split(" ");
                        data.article.categoryData.push({
                            "categoryType": categoryWithCount[1],
                            "count": categoryWithCount[0]
                        });

                    }
                });
                //console.log(`Category :${categoryWithCount[1]},Count:${categoryWithCount[0]}`);
                // console.log(data.article)
                //console.log(",")
                categoryResult.push(data.article);
                // Do not use index of first request as execution of request is asynchroneous so order of index is not correct.
                // console.log(totalCategory, index);
                console.log(totalCategory, executedCount);
                if (totalCategory == executedCount) {
// Step -3 Once All Request completed then Save Final Result to JSON File
                    console.log("When we reach at Last category then generate file for");
                    //When we reached at Last category request then generate file json file
                    fs.writeFile('cSharpcornerStatistics--' + new Date().toShortFormat() + '.json', JSON.stringify(categoryResult), "utf8", function(err) {

                        if (err) {
                            console.log(err)
                        };
                        console.log('Saved!,c-sharpcorner-statistics.json');
                    });
                }
            })

        }
    });
});

The full source code is also available in GitHub Repository here. Clone the above repository and run this app by running the below command as an administrator in Windows or as a sudo user in Linux.

• npm install (To install the dependency package)
• node app.js

The demo application dashboard using these C# Corner stats is available here.

 

This is a TUI Charts Library based Dashboard.

 

Conclusion

 

In this article, we learned different ways of generating C# Corner statistics, explored some problems and tools to achieve the desired data.