Web Scraping using C#

Question

Web Scraping using C#

Hi ,
I have been working on a project related to Web scraping using C# and have been able to scrap and crawl pages from the internet. But I am facing a challenge to scrap pages from website where authentication is required as my C# code is not able to clear the authentication even if I provide the ID and Password along with the URI in HTTPWebRequest.. Every time I am getting redirect to the login page and getting it as a response. Please advice ..

 

// My code sample 

string s = "http://MySite:82/DisplayUsers/"; 

HttpWebRequest wb = (HttpWebRequest) HttpWebRequest.Create(s);

NetworkCredential nc = new NetworkCredential();

nc.UserName = "User1";

nc.Password = "Password1";

CookieContainer cc = new CookieContainer();

wb.CookieContainer = cc;

CredentialCache myCache = new CredentialCache();

myCache.Add(new Uri(s),"Basic",nc);

myCache.Add(new Uri(s),"Digest", nc);

wb.Credentials = myCache;

wb.Method = "GET";

WebResponse response = wb.GetResponse();

Stream dataStream = response.GetResponseStream();

StreamReader reader = new StreamReader(dataStream);

string responseFromServer = reader.ReadToEnd();

 

Thanks ,

Roshan