Web Scraping using C#

Hi ,
I have been working on a project related to Web scraping using C# and have been able to scrap and crawl pages from the internet. But I am facing a challenge to scrap pages from website where authentication is required as my C# code is not able to clear the authentication even if I provide the ID and Password along with the URI in HTTPWebRequest.. Every time I am getting redirect to the login page and getting it as a response. Please advice ..

// My code sample

string s = "http://MySite:82/DisplayUsers/";

HttpWebRequest wb = (HttpWebRequest) HttpWebRequest.Create(s);

NetworkCredential nc = new NetworkCredential();

nc.UserName = "User1";

nc.Password = "Password1";

CookieContainer cc = new CookieContainer();

wb.CookieContainer = cc;

CredentialCache myCache = new CredentialCache();

myCache.Add(new Uri(s),"Basic",nc);

myCache.Add(new Uri(s),"Digest", nc);

wb.Credentials = myCache;

wb.Method = "GET";

WebResponse response = wb.GetResponse();

Stream dataStream = response.GetResponseStream();

StreamReader reader = new StreamReader(dataStream);

string responseFromServer = reader.ReadToEnd();

Thanks ,

Roshan

2 Replies

Know the answer? Post it — somebody with the same question will find it here.

Roshan Dash11y ago
Hi Manoj, Thanks for the response. Either way its not working.
Manoj Bhoir11y ago
Hi,

Why you are using CredentialCache Here. I think you can directly assign NetworkCredential to HttpWebRequest.Credentials Property.

Try this :
wb.Credentials = nc;
Instead Of
wb.Credentials = myCache;

I hope this may help you.