Run the urls in python
<p>I would like to download some datasets from a webpage X. Links to my datasets are included on the webpage. This is my code (I changed the name of the webpage)</p>

<pre><code>import urllib2
from BeautifulSoup import *
import webbrowser

link = urllib2.urlopen('http://www.x.com/html/tlc/html/about/abcd.shtml').read()
soup = BeautifulSoup(link)

tags = soup('a')

for tag in tags:
    if 'https://s3.amazonaws.com/abcd/qwer/' in tag.get('href'):
        webbrowser.open(tag.get('href'))
</code></pre>

<p>This works fine. However, there are around 40 links of datasets I need to download on the webpage. If I run this code, my Chrome will open 40 new tabs almost at the same time, and since my computer is quite slow, only 6 datasets are downloaded.</p>

<p>Really appreciate if someone could help me fix this</p>

<p>Since this seems to be a problem of your computer/browser, you could try to not download them all at once.</p>

<p>Have a look at this:
<a href="http://stackoverflow.com/questions/24533951/wait-for-download-completion-in-python">Wait for download completion in Python</a></p>

<p>This uses the urllib without opening the browser.</p>

python beautifulsoup