Downloading PDFs from links scraped with Beautiful Soup
<p>I'm trying to write a script which will iterate through a list of landing page urls from a csv file, append all PDF links on the landing page to a list, and then iterate through the list downloading the PDFs to a specified folder. </p>

<p>I'm a bit stuck on the last step- I can get all the PDF urls, but can only download them individually. I'm not sure how best to amend the directory address to change with each url to ensure each has its own unique file name. </p>

<p>Any help would be appreciated!</p>

<pre><code>from bs4 import BeautifulSoup, SoupStrainer
import requests
import re

#example url
url = "https://beta.companieshouse.gov.uk/company/00445790/filing-history"
link_list = []
r = requests.get(url)
soup = BeautifulSoup(r.content, "lxml")

for a in soup.find_all('a', href=True):
    if "document" in a['href']:
        link_list.append("https://beta.companieshouse.gov.uk"+a['href'])

for url in link_list:

    response = requests.get(url)

    with open('C:/Users/Desktop/CompaniesHouse/report.pdf', 'wb') as f:
        f.write(response.content)
</code></pre>

<p>The simplest thing  is to just add a number to each filename using enumerate:</p>

<pre><code>for ind, url in enumerate(link_list, 1):
    response = requests.get(url)

    with open('C:/Users/Desktop/CompaniesHouse/report_{}.pdf'.format(ind), 'wb') as f:
        f.write(response.content)
</code></pre>

<p>But presuming each path ends in  <em>somne_filename.pdf</em> and they are unique you can  use the <em>basename</em> itself which may be more descriptive:</p>

<pre><code>from os.path import basename, join
for url in link_list:  
    response = requests.get(url)   
    with open(join('C:/Users/Desktop/CompaniesHouse", basename(url)), 'wb') as f:
        f.write(response.content)
</code></pre>

python pdf beautifulsoup