Retrieve a list of files located at a URL with filenames matching a known pattern
<p>There is a URL where a colleague has set up a large number of files for me to download, </p>

<pre><code>url = "http://www.some.url.edu/some/dirname/"
</code></pre>

<p>Inside this directory, there are a large number of files with different filename patterns that are known to me in advance, e.g., "subvol1_file1.tar.gz", "subvol1_file2.tar.gz", etc. I am going to selectively download these files based on their filename patterns using fnmatch. </p>

<p>What I need is a simple list or generator of <em>all</em> filenames located in <em>dirname</em>. Is there a simple way to use, for example, BeautifulSoup or urllib2 to retrieve such a list? </p>

<p>Once I have the list/iterable, let's call it <strong>filename_sequence</strong>, I plan to download the files with a pattern <em>filepat</em> with the following pseudocode:</p>

<pre><code>filename_sequence = code_needed
filepat = "*my.pattern*"
import os, fnmatch
for basename in fnmatch.filter(filename_sequence, filepat):
    os.system("wget "+os.path.join(url, basename))
</code></pre>

<p>Not sure this is applicable in your case, but you can apply a regular expression pattern on the <code>href</code> attribute values:</p>

<pre><code>import re

pattern = re.compile(r"subvol1_file\d+\.tar\.gz")
links = [a["href"] for a in soup.find_all("a", href=pattern)]
</code></pre>

python beautifulsoup urllib2