Get all URLs accessed from webpage

Question

I have a website I want to extract a URL that is loaded (not in the mail html page source, and then do some processing with it. The URL would contact m1pl in it. I have been looking at fiddlercore and Selenium as I know when I load the page it shows up in Chrome under F12 then Network tab. But I need an automation as it's lot of links and runs all the time, So need a way to load the page, preferably one not on the screen, pull all the URLs it loads, and then find the one containing m1pl . Or something like that. I've been looking as mentioned at the above tools but haven't found the magic sauce. Plus Selenium seems to keep the Chrome browser open and never closes it. Thoughts? Thanks. JR

Sachin Singh · Answer

open any website on chrome , press f12 and then press Esc , in the console window paste below code, it will show you all links in that website, now you can use jquery or other to automate this. var x = document.querySelectorAll( "a" ); var myarray = [] for (var i=0; i var nametext = x[i].textContent; var cleantext = nametext.replace(/\s+/g, ' ' ).trim(); var cleanlink = x[i].href; myarray.push([cleantext,cleanlink]); }; function make_table() { var table = ' Name Links ' ; for (var i=0; i table += ' ' + myarray[i][0] + ' ' +myarray[i][1]+ ' ' ; }; var w = window.open( "" ); w.document.write(table); } make_table()

John Riker · Answer

Thanks, I'll take a look at this suggestion purely out of interst however I may not have explained my need fully but this needs to be automated. Could be used 100's of times with 100's of URLs so manually going into chrome dev tools and looking wouldn't really work. I could do this now without any code if I manually pulled it up with Chrome. Also tried the above to see what as returned and it seems like things are missing. I can find the URLs that contain the requested value by manually looking at the network tab but when I try the code in the console the results started out looking cool but none of those were there.

Get all URLs accessed from webpage

Answers (2)