Screen Scraping using System.Net

Shantanu

14y76k180100Article

HttpWebRequest HttpWebResponsec# Screen ScrapingScreen ScrapingScreen Scraping c#

18 Comments

Join the conversation! Your thoughts help the community grow.

ajit machhe9y ago
Its not working when i try to use linkedin ? could you plz assist the direction to look ?
Ahmar Husain13y ago
I think WebClient class and its DownloadString method is much easier to use .
asp learner13y ago
HI.. UR ARTICLE IS NICE.. could you tell me how to login to the webpage using httpwebrequest
polas anderson14y ago
Hi your code not working i do know why but it doea not work when i try to seek for emails it drops me this program turn off in the seconds. Why is this happining ?
Michael Brummett15y ago
Your example is very helpful but I am also not able to unzip the download.
- Kurt Bank15y ago
  Uh, I'm sorry, but that makes no sense. An extension should reflect the type of file it is. It's not a popularity contest. I don't have WinZip or WinRar anymore as I have no need for them. Since zip is now integrated into Windows Explorer I just use it. Simple is good. It doesn't support a rar, only zip. Surprisingly, it also doesn't support a rar file renamed to a zip?!?! Duh.
- Kurt Bank15y ago
  Exactly. Most is not all. Therefore, I see no reason to change the file extension. It should be .rar as that is the type of file it is. It would have saved me time by knowing it wouldn't work with Windows Explorer. That's why type extensions matter and why changing them to something that is wrong just doesn't make sense. No matter. It was still a well written article and thanks for it.
- Sam Hobbs15y ago
  I don't understand why this web site uses rar files. Since the Windows Explorer supports zip files but not rar files, I think only zip should be used. For those that are interested, 7-Zip will open rar files. http://sourceforge.net/projects/sevenzip
Kurt Bank15y ago
Liked the article. Like the title. Download file says it's invalid though.
- Kurt Bank15y ago
  Tried again this morning. Same issue. The download seems to work fine. No error.When I click on the download in Win7 Windows Explorer it gives an error "Windows cannot open the folder. The compressed (zipped) folder ....... is invalid". If I right click on it and extract all, get an error "folder is empty". The folder has a reported size of 63KB so it's not empty, but obviously Windows doesn't like something about it. I've downloaded a number of zipped files from this site over the past few months with no issues. Downloaded another article files yesterday with no problem, so I don't know what's going on with this particular file for me. Thought I'd report it in case others were having the same problem.
- Sam Hobbs15y ago
  I opened the zip file using the VS binary editor. The zip file is a rar file. Rename the downloaded file changing the extension to rar instead of zip.
- Shantanu15y ago
  Hi. The download is a renamed .rar file which opens up in WinZip too. I had renamed it to zip since zip is a more popular format. However, you can rename it to a .rar extension and open it in WinRar if zip is giving problems with your version of winzip. Thanks.
laks_win16y ago
Article is nice. Reckon, the titles "Email scraping" or "Web page content scraping" would be fine
- Sam Hobbs15y ago
  I agree that this is web page scraping or something such as that but not screen scraping. The data being scraped is written to a web page using web page techniques (HTML, DOM, ASP or whatever) and is accessed by the program using the DOM. The data is not written or read using screen I/O.
- Shantanu16y ago
  Thanks. Glad you liked it. As far as the name goes, Screen Scraping is the industry term for such work. I have built an Email scraper just as an example of how to use this technique in .Net

Purpose

Screen Scraping is used to extract data from a web page by scraping it instead of more direct access. It involves requesting the page and then parsing the response. It is useful in situations where direct access to the data is not there.

Implementation

Suppose we want to scrape Email addresses from live web sites. It would be a two-step process with first fetching the page and then parsing the response contents to extract the email addresses.

We have created a class called Extract (shown below) which has an API called GetPage() which fetches the page given a URL.

System.Net namespace contain classes which can be used to fetch the page. The classes we are going to use are HttpWebRequest and HttpWebResponse.

Below is the GetPage() API :

public string GetPage()

{

HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(this.URL.Trim());

request.ContentType = "text/html";

request.Method = "GET";

request.Proxy = WebProxy.GetDefaultProxy();

request.Proxy.Credentials = CredentialCache.DefaultCredentials;

string Input = string.Empty;

try

{

HttpWebResponse response = (HttpWebResponse)request.GetResponse();

if (response.StatusCode == HttpStatusCode.OK)

{

using (StreamReader sr = new StreamReader(response.GetResponseStream()))

{

Input = sr.ReadToEnd();

// Close and clean up the StreamReader

sr.Close();

}

catch (Exception ex) { }

return Input;

}

The next step is to extract the Email addresses from the returned page. This is done by using Regular Expressions in the ExtractEmails() API. The Regex used is not a complete solution to the Email pattern but works with most email addresses. It is only used to demonstrate the technique.

public const string ALLOWED_CHARS = @"[a-zA-Z0-9-_]";

public const string REGEX_EMAILS = @"^((?<emails>" + ALLOWED_CHARS + @"+(\." + ALLOWED_CHARS + @"+)*@" + ALLOWED_CHARS + @"+(\." + ALLOWED_CHARS + @"+)*)*(.|\n|\r\n)*?)+$";

public void ExtractEmails(string Input)

{

Match m = Regex.Match(Input, REGEX_EMAILS);

if (m.Success)

{

foreach (Capture c in m.Groups["emails"].Captures)

{

Emails.Add(c.Value.Trim());

}

Then the extracted Email addresses are displayed on the screen separated by the defined separator.

Below is a snapshot of the Win Email Extractor tool I have built to demonstrate the technique. As an example, we are scraping Email addresses of US Senators from a public site that hosts those Email addresses.

Below is link to the public site:

http://www.surrogacy.com/legals/senators.html

Before using the tool, see that the website can be accessed in the browser from the machine on which the tool is being run. Sometimes issues like Proxy, firewall can cause the tool to not work.