Scraping issues on a specific website
<p>This is my first question on stack overflow so bear with me, please. </p>

<p>I am trying to download automatically (i.e. scrape) the text of some Italian laws from the website: <a href="http://www.normattiva.it" rel="nofollow">http://www.normattiva.it/</a></p>

<p>I am using this code below (and similar permutations):</p>

<pre><code>import requests, sys

debug = {'verbose': sys.stderr}
user_agent = {'User-agent': 'Mozilla/5.0', 'Connection':'keep-alive'}

url = 'http://www.normattiva.it/atto/caricaArticolo?art.progressivo=0&amp;art.idArticolo=1&amp;art.versione=1&amp;art.codiceRedazionale=047U0001&amp;art.dataPubblicazioneGazzetta=1947-12-27&amp;atto.tipoProvvedimento=COSTITUZIONE&amp;art.idGruppo=1&amp;art.idSottoArticolo1=10&amp;art.idSottoArticolo=1&amp;art.flagTipoArticolo=0#art'

r = requests.session()
s = r.get(url, headers=user_agent)
#print(s.text)
print(s.url)
print(s.headers)
print(s.request.headers)
</code></pre>

<p>As you can see I am trying to load the "<strong>caricaArticolo</strong>" query. </p>

<p>However, the output is a page saying that my search is invalid (<strong><em>"session is not valid or expired"</em></strong>) </p>

<p>It seems that the page recognizes that I am not using a browser and loads a "breakout" javascript function. </p>

<pre><code>&lt;body onload="javascript:breakout();"&gt;
</code></pre>

<p>I tried to use "browser" simulator python scripts such as <strong>selenium</strong> and <strong>robobrowser</strong> but the result is the same. </p>

<p>Is there anyone who is willing to spend 10 minutes looking at the page output and give help?</p>

<p>wonderful, wonderful, wonderful Padraic. It works. Just had to edits slightly to clear imports but it's works wonderfully. Thanks very much. I am just discovering python's potential and you have made my journey much easier with this specific task. I would have not solved it alone. </p>

<pre><code>import requests, sys
import os
from urllib.parse import urljoin
from bs4 import BeautifulSoup
import io
user_agent = {'User-agent': 'Mozilla/5.0', 'Connection': 'keep-alive'}

url = 'http://www.normattiva.it/atto/caricaArticolo?art.progressivo=0&amp;art.idArticolo=1&amp;art.versione=1&amp;art.codiceRedazionale=047U0001&amp;art.dataPubblicazioneGazzetta=1947-12-27&amp;atto.tipoProvvedimento=COSTITUZIONE&amp;art.idGruppo=1&amp;art.idSottoArticolo1=10&amp;art.idSottoArticolo=1&amp;art.flagTipoArticolo=0#art'

with requests.session() as s:
    s.headers.update(user_agent)
    r = s.get("http://www.normattiva.it/")
    soup = BeautifulSoup(r.content, "lxml")
    # get all the links from the initial page
    for a in soup.select("div.testo p a[href^=http]"):
        soup = BeautifulSoup(s.get(a["href"]).content)
        # The link to the text is in a iframe tag retuened from the previous get.

        text_src_link = soup.select_one("#mainFrame")["src"]

        # Pick something to make the names unique
        with io.open(os.path.basename(text_src_link), "w", encoding="utf-8") as f:
            # The text is in pre tag that is in the  div with the pre class
            text = BeautifulSoup(s.get(urljoin("http://www.normattiva.it", text_src_link)).content, "html.parser")\
                .select_one("div.wrapper_pre pre").text
            f.write(text)
</code></pre>

python web-scraping python-requests