Python->BeautifulSoup->Webscraping->Drop Down Menu
<p>So I am trying to dump all of the reports from this website:</p>

<p><a href="https://www.treasurydirect.gov/govt/reports/tfmp/tfmp_utf.htm" rel="nofollow">https://www.treasurydirect.gov/govt/reports/tfmp/tfmp_utf.htm</a></p>

<p>State: All States (Not the Reed Act Benefit or Reed Act Admin)</p>

<p>Report: Transaction Statement</p>

<p>Month: All Months</p>

<p>Year:  All Years</p>

<p>Looking at the Source Code of the website, I know that the state variables:</p>

<pre><code>&lt;form action="get" name="UtfReport"&gt;
&lt;fieldset&gt;
 &lt;table&gt;
&lt;tr&gt;

    &lt;td&gt;
    &lt;label for="states"&gt;State&lt;/label&gt;&lt;br /&gt;
        &lt;select name="states" id="states" size="01"&gt;
        &lt;option value="al" selected&gt;Alabama&lt;/option&gt;
        &lt;option value="b2"&gt;Alabama Reed Act Benefit&lt;/option&gt;
        &lt;option value="b3"&gt;Alabama Reed Act Admin&lt;/option&gt;
        &lt;option value="ak"&gt;Alaska&lt;/option&gt;
        &lt;option value="a2"&gt;Alaska Reed Act Benefit&lt;/option&gt; 
</code></pre>

<p>So I know that I need to create a list of string like this</p>

<pre><code>https://www.treasurydirect.gov/govt/reports/tfmp/utf/[a1]/dfiw00[116]tsar.txt

https://www.treasurydirect.gov/govt/reports/tfmp/utf/[a1]/dfiw00[216]tsar.txt
</code></pre>

<p>....</p>

<p>So here is my current approach: </p>

<pre><code>import requests, bs4



for i in range(1,13):
    print('https://www.treasurydirect.gov/govt/reports/tfmp/utf/ar/dfiw0'+str(i).zfill(2),'16tsar.txt')


res = requests.get('https://www.treasurydirect.gov/govt/reports/tfmp/utf/ar/dfiw00216tsar.txt')

res.raise_for_status()
states = bs4.BeautifulSoup(res.text, 'lxml')

result1.append(res.text)
</code></pre>

<p>My effort in create the string of url ran into a problem as well, as this is the output from the code above (there is a space between dfiw00X and 16tsar.txt and I don't know why):</p>

<pre><code>https://www.treasurydirect.gov/govt/reports/tfmp/utf/ar/dfiw001 16tsar.txt
https://www.treasurydirect.gov/govt/reports/tfmp/utf/ar/dfiw002 16tsar.txt
https://www.treasurydirect.gov/govt/reports/tfmp/utf/ar/dfiw003 16tsar.txt
https://www.treasurydirect.gov/govt/reports/tfmp/utf/ar/dfiw004 16tsar.txt
https://www.treasurydirect.gov/govt/reports/tfmp/utf/ar/dfiw005 16tsar.txt
https://www.treasurydirect.gov/govt/reports/tfmp/utf/ar/dfiw006 16tsar.txt
https://www.treasurydirect.gov/govt/reports/tfmp/utf/ar/dfiw007 16tsar.txt
https://www.treasurydirect.gov/govt/reports/tfmp/utf/ar/dfiw008 16tsar.txt
https://www.treasurydirect.gov/govt/reports/tfmp/utf/ar/dfiw009 16tsar.txt
https://www.treasurydirect.gov/govt/reports/tfmp/utf/ar/dfiw010 16tsar.txt
https://www.treasurydirect.gov/govt/reports/tfmp/utf/ar/dfiw011 16tsar.txt
https://www.treasurydirect.gov/govt/reports/tfmp/utf/ar/dfiw012 16tsar.txt
</code></pre>

<p>So My Question is:  There must be a better way of doing this than the way I am currently trying, so if anyone can show me how, I would really appreciate it.</p>

<p>Thank you for your time,</p>

<h1>A good first approach to scraping: Pattern Matching Heuristic</h1>

<p>What you want to do, at a high level, is this:</p>

<ol>
<li>Identify a pattern in the source.</li>
<li>Represent the nature of the pattern in the code.</li>
<li>Scrape according to that code.</li>
</ol>

<h3>I won't code the entire thing here, but outline the general approach I would take.</h3>

<p>A. Notice that there's a pattern to the way the reports are named. If there is a pattern, then we can assume it is possible to represent it in code.</p>

<p>B. Of primary interest is the last part of the url, <code>'/ar/dfiw00216tsar.txt'.</code></p>

<ol>
<li>/ar/ references the state</li>
<li>dfiw appears constant, at first glance</li>
<li>00216 references the date</li>
<li>tsar references the type of report</li>
<li>.txt appears constant, at first glance</li>
</ol>

<p>From here, we can know to build a dictionary of all the possible states, and a dictionary of all the possible report types, and iterate through all of those combinations, including date, in each iteration of the for loop, getting the url, then saving it or otherwise processing it as needed.</p>

python drop-down-menu web-scraping beautifulsoup