Scraping Web site Dynamic Data using WATIN

Users can also un-comment this code snippet.

// ieInstance.Visible = false;
// This will wait for the browser to complete loading of the page
ieInstance.WaitForComplete();
// This will store page source in categoryPageSource variable
string categoryPageSource = ieInstance.Html;

Using Regular Expression features (RegEx and MatchCollection) of .NET to fetch respective data from the HTML page source

I have used regular expressions for fetching categories and to do iterative logic to fetch items in the respective categories and to move to the next page using regular expressions for fetching all the pages for the respective category items.

Please refer to the following regular expressions used for Category, Item fetching, and page navigation respectively.

1. Category Regular Expression: The following regular expression will fetch all the URL categories from the CategoryListing.aspx page and will navigate in a recursive loop.

   <A\S.*?class=bold\s.*?href="(?<href>.*?)"></span>

2. Item Regular Expression: The following regular expression will fetch ProductID, Product Name, and Product Price for the respective item residing in the given page.

   <P\s*id=.*?>ProductID:\s*<B>(?<ProductID>.*?)</B>.*?</P>\s*.*?<P\s*id=.*?>ProductName:\s*<B>(?<ProductName>.*?)</B>.*?</P>\s*.*?<P\s*id=.*?>ProductPrice:\s*<B>(?<ProductPrice>.*?)</B>.*?</P>

3. Paging Regular Expression: The following regular expression will fetch respective pages from the ItemListing.aspx page.

   (?(?=<SPAN>.*?</SPAN>)<SPAN>(?<PageNumber>.*?)\s*</SPAN>|<A\s*href="javascript.*?>(?<PageNumber>.*?)\s*</A>)
   

To use this regular expression in this application, I have used the "RegEx" class of the "System.Text.RegularExpression" namespace. RegEx will compile respective regular expression patterns using various options like "RegexOptions.Compiled", "RegexOptions.IgnoreCase", "RegexOptions.IgnorePatternWhitespace" and "RegexOptions.CultureInvariant".

Refer to the following code snippet for that.

// Regular expression for Category listing page
private const string _categoryRegEx = @"<A\S.*?class=bold\s.*?href=""(?<href>.*?)""></span>";
Regex categoryMatches = new Regex(_categoryRegEx, RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.IgnorePatternWhitespace | RegexOptions.CultureInvariant);

To fetch records based upon regular expression it requires using MatchCollection to fetch a list of successful matches for the respective

HTML Source generated (string categoryPageSource = ieInstance.Html;) as per #2 above. Refer to the following code for reference.

MatchCollection categoryMatchCollection = categoryMatches.Matches(categoryPageSource);

Use a for loop to fetch the respective results of a single match. Refer to the following code for reference.

foreach (Match categoryMatch in categoryMatchCollection)

If a regular expression is created based upon the group, then use the GroupCollection method to fetch groups of the respective results. Refer to the following code for reference.

GroupCollection categoryGroup = categoryMatch.Groups;

A Group collection contains multiple associated groups. To fetch a category, I have used "href" as a group. Refer to the following code for reference.

string itemListingURL = Convert.ToString(categoryGroup["href"].Value);

Now, the itemListingURL variable will contain the href for the respective category. Now Watin will navigate to this URL as in the following.

The itemListingPath variable contains the full path of the item listing page for the respective category.

ieInstance.GoTo(itemListingpath);

I have used the "WaitForComplete" method to wait until the respective page has loaded completely. Refer to the following code for reference.

ieInstance.WaitForComplete();

Using the code above, the application will navigate to the item listing page. A similar operation needs to be performed for fetching items.

Once all items of the respective page are fetched and navigation to the next page has been done, Watin provides a Click event to do click on a specific page.

The Click event can also be performed based on other criteria.

Please refer to the following

You can get more information on all the preceding criteria by visiting http://watin.org.

In this article, I have used "Find.ByText" to find a link by text and then perform the click event. You can also attach a regular expression with the above criteria.

Refer to the following code for reference.

// Fetches the page number of the current page.
string linkText = Convert.ToString(pagingGroup[_pageNumber].Value);
// Performs click event on the given link. For e.g, if linkText contains "2" as a value then Watin will perform a click event on this second link.
ieInstance.Link(Find.ByText(linkText)).Click();
// Wait for the operation to complete
ieInstance.WaitForComplete();
// Store the result of the page in the itemListingPageSource variable
itemListingPageSource = ieInstance.Html;

Once the respective items in the web page are scraped, the current application will store respective items in "Output.txt" using StreamWriter of System.IO namespace.

Now open the "Output.txt" file and observe that it contains all the items of Men, Women, and Children Categories.

Execute this application using NUnit

Execute the WatinWebScraper application, requires doing the following procedure.

1. Open the NUnit application.
2. Now click on "File" -- "Open Project" and navigate to the DLL file ("WatinWebScraping.dll") of the Watin Web Scraper application. Refer to the following image for reference.
3. Now click on the "Run" button as shown in the image above. Observe that the application will start scraping the Demo Web application by Navigating to the Category and all its respective items will be stored in the "Output.txt" file.