How to grab content from webpage source?

John Riker

Tried posting this to my other thread but it wouldn't post.

I am trying to figure out how to pull content from fox.com and a Comedy Central site and their API's are either protected, or need an API Key but no way to get one, or need some kind of token I don't get so give up. Not enough detail out there to figure them out. Anyone know say for these two sites how I would pull out a list of shows, their associated metadata info like series and episode number, and also the link to their video page?

https://www.fox.com/the-masked-singer/

So the rub on this one, I would want the first four videos but not the stuff below it like the Clips and Extras. And as you can see there is a scroller so any way to get videos on the next "page"?

http://www.cc.com/shows/the-daily-show-with-trevor-noah

With this one I'd want the Full Episodes and again the scroller to older content.

Thoughts?

Thanks.

2 Replies

Know the answer? Post it — somebody with the same question will find it here.

Virender Verma5y ago
By the way, one good example from C-Sharpcorner is as follows for data scrapping. It basically scraps the data from the webpage and you can tweak this to fit your requirements of scrapping content from above mentioned 2 sites.

https://www.c-sharpcorner.com/article/data-scraping-in-a-webpage-automation-using-uipath-studio/
Virender Verma5y ago
Hi,
I think you can apply data/web scrapping concept here. There are multiple tools available already to meet your needs. You can explore and take it forward as per your need.

Do let us know in case this doesn't meet your needs.

Thanks and cheers.