Scraping table information
<p>I would like to extract the odds information from the website <a href="http://www.footballlocks.com/nfl_odds.shtml" rel="nofollow">http://www.footballlocks.com/nfl_odds.shtml</a> using Python.</p>

<p>I have been trying to do it with BeautifulSoup.</p>

<p>The optimal result would be to obtain the odds information in dictionary or list format, as the values will be fed into a mathematical formula.</p>

<p>the HTML code for the odds information is:</p>

<pre><code>    &lt;TABLE COLS="6" WIDTH="650" BORDER="0" CELLSPACING="5" CELLPADDING="2"&gt;

  &lt;TR&gt;
    &lt;TD WIDTH="19%"&gt;&lt;span title="Date and Time of Game."&gt;&lt;B&gt;Date &amp; Time&lt;/B&gt;&lt;/span&gt;&lt;/TD&gt;
    &lt;TD WIDTH="21%"&gt;&lt;span title="Team Spotting Points in a Bet Against the Point Spread."&gt;&lt;B&gt;Favorite&lt;/B&gt;&lt;/span&gt;&lt;/TD&gt;
    &lt;TD WIDTH="14%"&gt;&lt;span title="Short for Point Spread. Number of Points Subtracted from Final Score of Favorite to Determine Winner of a Point Spread Based Wager."&gt;&lt;B&gt;Spread&lt;/B&gt;&lt;/span&gt;&lt;/TD&gt;
    &lt;TD WIDTH="21%"&gt;&lt;span title="Team Receiving Points in a Bet With the Point Spread."&gt;&lt;B&gt;Underdog&lt;/B&gt;&lt;/span&gt;&lt;/TD&gt;
    &lt;TD WIDTH="6%"&gt;&lt;span title="Line for Betting Over or Under the Total number of Points Scored by Both Teams Combined. Synonymous With Over/Under."&gt;&lt;B&gt;Total&lt;/B&gt;&lt;/span&gt;&lt;/TD&gt;
    &lt;TD WIDTH="19%"&gt;&lt;span title="Money odds to Win the Game Outright, Without any Point Spread. 
Minus (-) is Amount Bettors Risk for Each $100 on the Favorite to Win the Game Outright.
Plus (+) is Amount Bettors Win for Each $100 Risked on the Underdog to Win the Game Outright."&gt;&lt;B&gt;Money Odds&lt;/B&gt;&lt;/span&gt;&lt;/TD&gt;
  &lt;/TR&gt;






 &lt;TR&gt;
    &lt;TD&gt;9/18 1:00 ET&lt;/TD&gt;
    &lt;TD&gt;At Detroit&lt;/TD&gt;
    &lt;TD&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;-6&lt;/TD&gt;
    &lt;TD&gt;Tennessee&lt;/TD&gt;
    &lt;TD&gt;47&lt;/TD&gt;
    &lt;TD&gt;-$255 +$215&lt;/TD&gt;
  &lt;/TR&gt;

 &lt;TR&gt;
    &lt;TD&gt;9/18 1:00 ET&lt;/TD&gt;
    &lt;TD&gt;At Houston&lt;/TD&gt;
    &lt;TD&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;-2.5&lt;/TD&gt;
    &lt;TD&gt;Kansas City&lt;/TD&gt;
    &lt;TD&gt;43&lt;/TD&gt;
    &lt;TD&gt;-$140 +$120&lt;/TD&gt;
  &lt;/TR&gt;

 &lt;TR&gt;
    &lt;TD&gt;9/18 1:00 ET&lt;/TD&gt;
    &lt;TD&gt;At New England&lt;/TD&gt;
    &lt;TD&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;-6.5&lt;/TD&gt;
    &lt;TD&gt;Miami&lt;/TD&gt;
    &lt;TD&gt;42&lt;/TD&gt;
    &lt;TD&gt;-$290 +$240&lt;/TD&gt;
  &lt;/TR&gt;

 &lt;TR&gt;
    &lt;TD&gt;9/18 1:00 ET&lt;/TD&gt;
    &lt;TD&gt;Baltimore&lt;/TD&gt;
    &lt;TD&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;-6.5&lt;/TD&gt;
    &lt;TD&gt;At Cleveland&lt;/TD&gt;
    &lt;TD&gt;42.5&lt;/TD&gt;
    &lt;TD&gt;-$300 +$250&lt;/TD&gt;
  &lt;/TR&gt;

 &lt;TR&gt;
    &lt;TD&gt;9/18 1:00 ET&lt;/TD&gt;
    &lt;TD&gt;At Pittsburgh&lt;/TD&gt;
    &lt;TD&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;-3.5&lt;/TD&gt;
    &lt;TD&gt;Cincinnati&lt;/TD&gt;
    &lt;TD&gt;48.5&lt;/TD&gt;
    &lt;TD&gt;-$180 +$160&lt;/TD&gt;
  &lt;/TR&gt;

 &lt;TR&gt;
    &lt;TD&gt;9/18 1:00 ET&lt;/TD&gt;
    &lt;TD&gt;At Washington&lt;/TD&gt;
    &lt;TD&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;-2.5&lt;/TD&gt;
    &lt;TD&gt;Dallas&lt;/TD&gt;
    &lt;TD&gt;45.5&lt;/TD&gt;
    &lt;TD&gt;-$145 +$125&lt;/TD&gt;
  &lt;/TR&gt;

 &lt;TR&gt;
    &lt;TD&gt;9/18 1:00 ET&lt;/TD&gt;
    &lt;TD&gt;At NY Giants&lt;/TD&gt;
    &lt;TD&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;-4.5&lt;/TD&gt;
    &lt;TD&gt;New Orleans&lt;/TD&gt;
    &lt;TD&gt;53.5&lt;/TD&gt;
    &lt;TD&gt;-$225 +$185&lt;/TD&gt;
  &lt;/TR&gt;

 &lt;TR&gt;
    &lt;TD&gt;9/18 1:00 ET&lt;/TD&gt;
    &lt;TD&gt;At Carolina&lt;/TD&gt;
    &lt;TD&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;-13.5&lt;/TD&gt;
    &lt;TD&gt;San Francisco&lt;/TD&gt;
    &lt;TD&gt;45&lt;/TD&gt;
    &lt;TD&gt;-$900 +$600&lt;/TD&gt;
  &lt;/TR&gt;

 &lt;TR&gt;
    &lt;TD&gt;9/18 4:05 ET&lt;/TD&gt;
    &lt;TD&gt;At Arizona&lt;/TD&gt;
    &lt;TD&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;-7&lt;/TD&gt;
    &lt;TD&gt;Tampa Bay&lt;/TD&gt;
    &lt;TD&gt;50&lt;/TD&gt;
    &lt;TD&gt;-$310 +$260&lt;/TD&gt;
  &lt;/TR&gt;

 &lt;TR&gt;
    &lt;TD&gt;9/18 4:05 ET&lt;/TD&gt;
    &lt;TD&gt;Seattle&lt;/TD&gt;
    &lt;TD&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;-6.5&lt;/TD&gt;
    &lt;TD&gt;At Los Angeles&lt;/TD&gt;
    &lt;TD&gt;38&lt;/TD&gt;
    &lt;TD&gt;-$290 +$240&lt;/TD&gt;
  &lt;/TR&gt;

 &lt;TR&gt;
    &lt;TD&gt;9/18 4:25 ET&lt;/TD&gt;
    &lt;TD&gt;At Denver&lt;/TD&gt;
    &lt;TD&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;-6.5&lt;/TD&gt;
    &lt;TD&gt;Indianapolis&lt;/TD&gt;
    &lt;TD&gt;46.5&lt;/TD&gt;
    &lt;TD&gt;-$280 +$240&lt;/TD&gt;
  &lt;/TR&gt;

 &lt;TR&gt;
    &lt;TD&gt;9/18 4:25 ET&lt;/TD&gt;
    &lt;TD&gt;At Oakland&lt;/TD&gt;
    &lt;TD&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;-4.5&lt;/TD&gt;
    &lt;TD&gt;Atlanta&lt;/TD&gt;
    &lt;TD&gt;49&lt;/TD&gt;
    &lt;TD&gt;-$210 +$180&lt;/TD&gt;
  &lt;/TR&gt;

 &lt;TR&gt;
    &lt;TD&gt;9/18 4:25 ET&lt;/TD&gt;
    &lt;TD&gt;At San Diego&lt;/TD&gt;
    &lt;TD&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;-3&lt;/TD&gt;
    &lt;TD&gt;Jacksonville&lt;/TD&gt;
    &lt;TD&gt;47&lt;/TD&gt;
    &lt;TD&gt;-$165 +$145&lt;/TD&gt;
  &lt;/TR&gt;


 &lt;TR&gt;
    &lt;TD&gt;9/18 8:30 ET&lt;/TD&gt;
    &lt;TD&gt;Green Bay&lt;/TD&gt;
    &lt;TD&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;-2.5&lt;/TD&gt;
    &lt;TD&gt;At Minnesota&lt;/TD&gt;
    &lt;TD&gt;43.5&lt;/TD&gt;
    &lt;TD&gt;-$140 +$120&lt;/TD&gt;
  &lt;/TR&gt;




&lt;/TABLE&gt;
</code></pre>

<p>The Python code thus far.</p>

<pre><code>from bs4 import BeautifulSoup
import urllib

url = "http://www.footballlocks.com/nfl_odds.shtml"
html = urllib.urlopen(url)

soup = BeautifulSoup(html, 'html.parser')

for record in soup.find_all('tr'):
  for data in record.find_all('td'):
    print data.text
</code></pre>

<p>PS. My background is economics and my programming experience is limited.</p>

<p>It is not the nicest html to parse as there are no classes we can use but this will put all the rows into a list of dicts:</p>

<pre><code>from bs4 import BeautifulSoup
import requests


url = "http://www.footballlocks.com/nfl_odds.shtml"

soup = BeautifulSoup(requests.get(url).content)

# Use the text of one of the headers to find the correct table
table = soup.find("span", text="Date &amp; Time").find_previous("table")



data = []
# start from second tr 
for row in table.select("tr + tr"):
    # index to get the tds we need
    tds = [td.text for td in row.find_all("td")]
    fav, under, odds = tds[1], tds[2], tds[-1]
    # split money odds into fav/under odds
    f_odds,u_odds = odds.split()

    data.append({fav: f_odds.replace(u"$", ""), under : u_odds.replace(u"$", "")})
from pprint import pprint as pp
pp(data)
</code></pre>

<p>Output:</p>

<pre><code>[{u'At Detroit': u'-255', u'Tennessee': u'+215'},
 {u'At Houston': u'-130', u'Kansas City': u'+110'},
 {u'At New England': u'-290', u'Miami': u'+240'},
 {u'At Cleveland': u'+225', u'Baltimore': u'-265'},
 {u'At Pittsburgh': u'-175', u'Cincinnati': u'+155'},
 {u'At Washington': u'-150', u'Dallas': u'+130'},
 {u'At NY Giants': u'-215', u'New Orleans': u'+180'},
 {u'At Carolina': u'-900', u'San Francisco': u'+600'},
 {u'At Arizona': u'-330', u'Tampa Bay': u'+270'},
 {u'At Los Angeles': u'+250', u'Seattle': u'-300'},
 {u'At Denver': u'-275', u'Indianapolis': u'+235'},
 {u'At Oakland': u'-210', u'Atlanta': u'+180'},
 {u'At San Diego': u'-160', u'Jacksonville': u'+140'},
 {u'At Minnesota': u'+115', u'Green Bay': u'-135'}]
</code></pre>

python html web-scraping