Website Recursive Url Parser

Sign in to leave a comment.

• HV

  Haswell VA9y agoPosted Feb 15, 2017, 4:56 AM

  I have a mather... The Url start by http:/ and for use the url with the method URI it need to be http://. How can I correct it ?
• RJ

  rosepriya john11y agoPosted Sep 9, 2015, 5:25 AM

  Good work. Nice coding and helpful also...
• MA

  Mitul Ahir12y agoPosted Aug 14, 2014, 3:35 PM

  Can you please advise me how I can do similar things with HTML file? I have an HTML that has a hyperlink which points to another hyperlink, and that hyperlink points to some other. So I have so many levels but in folder, not on website. So there is no http:'// in hyperlink part. For example: href=123.html
• SP

  S Parui13y agoPosted Nov 18, 2012, 8:57 AM

  http://www.planetexim.net..this is my website
• 

  Jean Paul13y agoPosted Nov 16, 2012, 12:23 PM

  Which is your web site url? Send me the details to jeanpaulvaATgmailDOTTTTTcom Please send me the non-included url too. Thanks Jean Paul
• SP

  S Parui13y agoPosted Nov 16, 2012, 2:04 AM

  I am not able to get all url list of my site.Its returning only 40 urls of my site. But there are more than 2000 pages in my site. So please tell me how to get all the url list. Thanks in advance.
• S

  sachin14y agoPosted Oct 7, 2012, 8:37 AM

  let me try working with your application. Dude I need your expertise. Can you email me to [email protected]. I am waiting for your reply.
• 

  Jean Paul14y agoPosted Aug 3, 2012, 9:26 AM

  Yeah! good idea.. you can add new Thread(new ThreadStart()) for this or BackgroundWorker
• YF

  yunus farooqui14y agoPosted Aug 3, 2012, 7:22 AM

  Thanks for sharing . It would be helpful if you give idea to implement the logic with multithreading so tht execution would be faster thanks.
• YF

  yunus farooqui14y agoPosted Aug 3, 2012, 7:20 AM

  It is really nice program . It would be very helpful if you give idea to write this programe using multithreading so that execution would be much faster Thanks
• 

  Jean Paul14y agoPosted Jul 24, 2012, 9:16 AM

  You're Welcome Veeekay
• VP

  Veekay Pandey14y agoPosted Jul 24, 2012, 2:58 AM

  It is really nice program. But I am actually searching for the contact information extraction from different web sites. It will be really nice if you provide some information about that. Thank you in advance..
• 

  Sam Hobbs15y agoeditedPosted Mar 28, 2011, 4:23 PMEdited Mar 28, 2011, 4:26 PM

  I know that many people use RegularExpressions to parse HTML. Thank you for showing how to get all the urls in a website using RegularExpressions. I prefer to use the DOM though. The HTML can be put in a HtmlDocument and the Links Property can be used to get all the links in a web page. A lot of your code, such as recursion, is still useful even if the HtmlDocument class was used instead of RegularExpressions.
  • 

    Jean Paul15y agoPosted Mar 28, 2011, 10:36 PM

    That is a cool idea Sam to use HTML DOM. I found that the regular expression is failing when given a particular website. I cannot find the reason. So in this case we should be able to run over with HTML DOM. Thanks for sharing the info. I will compare the speed of both as well.
  • 

    Jean Paul15y agoPosted Mar 29, 2011, 12:09 AM

    Thanks a lot Sam. I prefer using the shldoc library as the SpiderLogic is a class library. Surely I will refer your articles. Thanks a lot for helping me. Have a nice day.
  • 

    Sam Hobbs15y agoPosted Mar 28, 2011, 11:44 PM

    Very good. So I think what you can do is to create an InternetExplorer object and put the HTML in there. If you are not using Windows Forms then I think you can use the classes and interfaces in the ShlDocVw library or something like that. There are some articles in this web site that will help with that, including my article.