extract data from website using python
<p>I recently started learning python and one of the first projects I did was to scrap updates from my son's classroom web page and send me notifications that they updated the site.  This turned out to be an easy project so I wanted to expand on this and create a script that would automatically check if any of our lotto numbers hit. Unfortunately I haven't been able to figure out how to get the data from the website. Here is one of my attempts from last night.</p>

<pre><code>from bs4 import BeautifulSoup
import urllib.request

webpage = "http://www.masslottery.com/games/lottery/large-winningnumbers.html"

websource = urllib.request.urlopen(webpage)
soup = BeautifulSoup(websource.read(), "html.parser")

span = soup.find("span", {"id": "winning_num_0"})
print (span)

Output is here...
&lt;span id="winning_num_0"&gt;&lt;/span&gt; 
</code></pre>

<p>The output listed above is also what I see if I "view source" with a web browser.  When I "inspect Element" with the web browser I can see the winning numbers in the inspect element panel.  Unfortunately I'm not even sure how/where the web browser is getting the data. is it loading from another page or a script in the background? I thought the following tutorial was going to help me but I wasn't able to get the data using similar commands. </p>

<p><a href="http://zevross.com/blog/2014/05/16/using-the-python-library-beautifulsoup-to-extract-data-from-a-webpage-applied-to-world-cup-rankings/" rel="nofollow">http://zevross.com/blog/2014/05/16/using-the-python-library-beautifulsoup-to-extract-data-from-a-webpage-applied-to-world-cup-rankings/</a></p>

<p>Any help is appreciated.
Thanks</p>

<p>If you look closely at the source of the page (I just used <code>curl</code>) you can see this block</p>

<pre><code>&lt;script type="text/javascript"&gt;
    // &lt;![CDATA[
    var dataPath = '../../';
    var json_filename = 'data/json/games/lottery/recent.json';
    var games = new Array();
    var sessions = new Array();
    // ]]&gt;
&lt;/script&gt;
</code></pre>

<p>That <code>recent.json</code> stuck out like a sore thumb (I actually missed the <code>dataPath</code> part at first).</p>

<p>After giving that a try, I came up with this:</p>

<pre><code>curl http://www.masslottery.com/data/json/games/lottery/recent.json
</code></pre>

<p>Which, as lari points out in the comments, is way easier than scraping HTML. This easy, in fact:</p>

<pre><code>import json
import urllib.request
from pprint import pprint

websource = urllib.request.urlopen('http://www.masslottery.com/data/json/games/lottery/recent.json')
data = json.loads(websource.read().decode())
pprint(data)
</code></pre>

<p><code>data</code> is now a dict, and you can do whatever kind of dict-like things you'd like to do with it. And good luck ;)</p>

python python-3.x