I am planning to create my own search engine .for that i have idea to create a crawler/bot that indexes the web but don't have idea where to host it so it can index web 24/7 and add new pages automatically to database ..
I prefer c# as language to code it .
also i want guidance on which type of database will be better for crawl engine Queue .
any idea about till which level I have to crawl any link ?
Loading
Mahesh ChandPosted May 31, 2010, 11:01 PM
You can definitely use MySQL as a database. If you use SQL Server, you will have to pay thousands $$$ to buy a production license. As per language goes, I think you will have to use more than C#. You definitely will need lot of JavaScript, C++, or some other languages. I am not sure just alone C# can do this work. You will also need network and socket programming.
Did you look at these samples?
http://www.codeproject.com/KB/IP/Crawler.aspx
How To: Write a Web Crawler in C# - The Coding Humanist
Cheers!
Kirtan PatelPosted May 31, 2010, 4:27 PM
some how it helped me but Lucene is not integrating with sql server no doubt lucene is good in query etc ... is there any better crawler that can do that ??
jingePosted May 28, 2010, 10:34 AM
Kirtan PatelPosted May 25, 2010, 4:53 PM
Mahesh ChandPosted May 25, 2010, 2:43 PM
To be frank, I have not done this kind (crawling) of work and would not have any idea. I have built many commerial applications and websites but have not touched this topic.