WebCrawler - Cookie Manegment

20y5.7k

Long time reader, first time I actually had to register and post: Simply put, I’m a college student that wants to create a quick web crawler for a research paper for a 400 level class. The issue is even with my experience in C# and the .NET architecture, I cannot work myself around a problem: There seems to be a lot of sites now that as soon as you “grab” (stream it or download or any other way) a web page, they reply by giving you the dreadful “Your are using an incompatible browser.” Any way I do it using a WebClient class or HttpWebResponse class, it never works. I’m not looking for any code here, just a general suggestion or idea for a solution to make my crawler look as if it’s a normal browser such as IE. Any help at all would be greatly appreciated.

2 Replies

Know the answer? Post it — somebody with the same question will find it here.

Preethi Vinayak Ponangi17y ago
Hmm ... what you could do is use the .NET AxwebBrowser component.
This gives you access to an internet explorer web browser and that in turn shouldn't give you any problems with the web browser compatibility. Works for me. Try it out.
But I agree that HttpWebRequest and response give you a quicker result while crawling because there's no UI component involved.

Thanks.
Vinayak Ponangi.
J20y ago
Let me simplify: Is there a way to make a web crawler using C# and .net look like a normal browser to a server? If so, how, no code needed just point to an article.