Python: Get javascript file from href tag of html
<p>Consider a website similar to this one:</p>

<p><a href="http://a810-bisweb.nyc.gov/bisweb/COsByLocationServlet?requestid=1&amp;allbin=3055311" rel="nofollow">http://a810-bisweb.nyc.gov/bisweb/COsByLocationServlet?requestid=1&amp;allbin=3055311</a></p>

<p>As one can see, the website contains links to pdf files referenced by an href tag in the page source, e.g.:</p>

<pre><code>&lt;a href="javascript:$('form_cofo_pdf_view_B000114563.PDF').submit();"&gt;B000114563.PDF&lt;/a&gt;
</code></pre>

<p>I would like to open the underlying file using python, effectively scraping the results.</p>

<pre><code>req = urllib2.Request("link.com")
page = urllib2.urlopen(req)
soup = BeautifulSoup(page)

links = []
for link in soup.findAll('a'):
    links.append(link.get("href"))
</code></pre>

<p>Normally I would just connect the base url with the href url to get the documents, but here, they are referenced with javascript. Hence I am not entirely sure how to access the files.</p>

<p>I would prefer to use urrlib2 and BeautifulSoup and not switch to Selenium to click on links. Does anyone have an idea to accomplish that? It would be greatly appreciated.</p>

<p>I downloaded few files and compared direct link with its name and all elements required in link you have in filename</p>

<p>Filename:</p>

<pre><code>form_cofo_pdf_view_B000114563.PDF
</code></pre>

<p>Direct link:</p>

<pre><code>http://a810-bisweb.nyc.gov/bisweb/CofoDocumentContentServlet
?passjobnumber=null
&amp;cofomatadata1=cofo
&amp;cofomatadata2=B
&amp;cofomatadata3=000
&amp;cofomatadata4=114000
&amp;cofomatadata5=B000114563.PDF
</code></pre>

<p>So you can create direct link when you get filename from string <code>javascript:$('form_cofo_pdf_view_B000114563.PDF').submit();</code></p>

<p>Working code: <a href="http://pastebin.com/kt72GSyYa" rel="nofollow">http://pastebin.com/kt72GSyYa</a></p>

javascript python html web web-scraping