I'm looking for a way to retrieve the ad URLs for this website. http://www.quiltingboard.com/resources/

What I want to do is probably write a script to continuously refresh the page and grab the ad URLs.

Any suggestion?

BeautifulSoup alone isn't going to cut it. The ads are injected via javascript ( they're doubleclick ads ).

Your options are:

• script something like selenium to look for the urls 10-15seconds after page load

• if you stay in pure python, you'll need to :

  1. make the initial request to parse with beautiful soup
  2. figure out what google was going to inject with javascript
  3. make a secondary request to doubleclick for the iframe or payload url

Those methods will only get you the doubleclick urls that handle conversion tracking. If you want to find out where they redirect to, you'll need to open those urls to discover their redirects.