Website Recursive Url Parser

Jean Paul

C#.NET VS 2010InternalGetUrls()Url ParserWebsite RecursiveWebsite Recursive Url Parser

16 Comments

Join the conversation! Your thoughts help the community grow.

Haswell VA9y ago
I have a mather... The Url start by http:/ and for use the url with the method URI it need to be http://. How can I correct it ?
rosepriya john11y ago
Good work. Nice coding and helpful also...
Mitul Ahir12y ago
Can you please advise me how I can do similar things with HTML file? I have an HTML that has a hyperlink which points to another hyperlink, and that hyperlink points to some other. So I have so many levels but in folder, not on website. So there is no http:'// in hyperlink part. For example: href=123.html
S Parui14y ago
http://www.planetexim.net..this is my website
Jean Paul14y ago
Which is your web site url? Send me the details to jeanpaulvaATgmailDOTTTTTcom Please send me the non-included url too. Thanks Jean Paul
S Parui14y ago
I am not able to get all url list of my site.Its returning only 40 urls of my site. But there are more than 2000 pages in my site. So please tell me how to get all the url list. Thanks in advance.
sachin14y ago
let me try working with your application. Dude I need your expertise. Can you email me to [email protected]. I am waiting for your reply.
Jean Paul14y ago
Yeah! good idea.. you can add new Thread(new ThreadStart()) for this or BackgroundWorker
yunus farooqui14y ago
Thanks for sharing . It would be helpful if you give idea to implement the logic with multithreading so tht execution would be faster thanks.
yunus farooqui14y ago
It is really nice program . It would be very helpful if you give idea to write this programe using multithreading so that execution would be much faster Thanks
Jean Paul14y ago
You're Welcome Veeekay
Veekay Pandey14y ago
It is really nice program. But I am actually searching for the contact information extraction from different web sites. It will be really nice if you provide some information about that. Thank you in advance..
Sam Hobbs15y agoedited
I know that many people use RegularExpressions to parse HTML. Thank you for showing how to get all the urls in a website using RegularExpressions. I prefer to use the DOM though. The HTML can be put in a HtmlDocument and the Links Property can be used to get all the links in a web page. A lot of your code, such as recursion, is still useful even if the HtmlDocument class was used instead of RegularExpressions.
- Jean Paul15y ago
  That is a cool idea Sam to use HTML DOM. I found that the regular expression is failing when given a particular website. I cannot find the reason. So in this case we should be able to run over with HTML DOM. Thanks for sharing the info. I will compare the speed of both as well.
- Jean Paul15y ago
  Thanks a lot Sam. I prefer using the shldoc library as the SpiderLogic is a class library. Surely I will refer your articles. Thanks a lot for helping me. Have a nice day.
- Sam Hobbs15y ago
  Very good. So I think what you can do is to create an InternetExplorer object and put the HTML in there. If you are not using Windows Forms then I think you can use the classes and interfaces in the ShlDocVw library or something like that. There are some articles in this web site that will help with that, including my article.

In this article I am trying to share a piece of code that might be useful to some of the developers.

We can find a lot of code in C# that will parse the http urls in given string. But it is difficult to find a code that will:

Accept a url as argument, parse the site content
Fetch all urls in the site content, parse the site content of each urls
Repeat the above process until all urls are fetched.

Scenario

Taking the website http://valuestocks.in (A Stock Market Site) as example I would like to get all the urls inside the website recursively.

Design

The main class is SpiderLogic which contains all necessary methods and properties.

The GetUrls() method is used to parse the website and return the urls. There are two overloads for this method.

The first one takes 2 arguments. The url and and a Boolean indicating if recursive parsing is needed or not.

E.g.: GetUrls(http://www.google.com", true);

The second one is 3 arguments, url, base url and recursive Boolean.

This method is intended for usage like the url is a sub level of the base url. And the web page contains relative paths. So in order to construct the valid absolute urls, the second argument is necessary.

E.g.: GetUrls("http://www.whereincity.com/india-kids/baby-names/ ", http://www.whereincity.com/ , true);

Method Body of GetUrls()

public IList<string> GetUrls(string url, string baseUrl,
bool recursive)
{
if (recursive)
{
_urls.Clear();
RecursivelyGenerateUrls(url, baseUrl);

return _urls;
}
else
return InternalGetUrls(url, baseUrl);
}

InternalGetUrls()

Another method of interest would be InternalGetUrls() which fetches the content of url, parses the urls inside it and constructs the absolute urls.

private IList<string> InternalGetUrls(string baseUrl, string absoluteBaseUrl)
{
IList<string> list = new List<string>();

Uri uri = null;
if (!Uri.TryCreate(baseUrl, UriKind.RelativeOrAbsolute, out uri))
return list;

// Get the http content
string siteContent = GetHttpResponse(baseUrl);

var allUrls = GetAllUrls(siteContent);

foreach (string uriString in allUrls)
{
uri = null;
if (Uri.TryCreate(uriString, UriKind.RelativeOrAbsolute, out uri))
{
if (uri.IsAbsoluteUri)
{
if (uri.OriginalString.StartsWith(absoluteBaseUrl)) // If different domain / javascript: urls needed exclude this check
{
list.Add(uriString);
}
}
else
{
string newUri = GetAbsoluteUri(uri, absoluteBaseUrl, uriString);
if (!string.IsNullOrEmpty(newUri))
list.Add(newUri);
}
}
else
{
if (!uriString.StartsWith(absoluteBaseUrl))
{
string newUri = GetAbsoluteUri(uri, absoluteBaseUrl, uriString);
if (!string.IsNullOrEmpty(newUri))
list.Add(newUri);
}
}
}

return list;
}

Handling Exceptions

There is an OnException delegate that can be used to get the exceptions occurring while parsing.

Tester Application

A tester windows application is included with the source code of the article.
You can try executing it.

The form accepts a base url as the input and clicking the Go button it parses the content of url and extracts all urls in it. If you need a recursive parsing please check the Is Recursive check box.

Next Part

In the next part of the article, I would like to create a url verifier website that verifies all the urls in a website. I agree after doing a search we can find free providers like that. My aim is to learn & develop a custom code that could be extensible and reusable across multiple projects by community.