BeautifulSoup webscraper issue: can't find certain divs/tables
<p>I'm having issues with scraping pro-football-reference.com. I'm trying to access the "Team Offense" table but can't seem to target the div/table.
The best I can do is:</p>

<pre><code>soup.find('div', {'id':'all_team_stats})
</code></pre>

<p>which doesn't return the table nor it's immediate div wrapper. The following attempts return "None":</p>

<pre><code>soup.find('div', {'id':'div_team_stats'})
soup.find('table', {'id':'team_stats'})
</code></pre>

<p>I've already scraped different pages simply by:</p>

<pre><code>soup.find('table', {'id':'table_id})
</code></pre>

<p>but I can't figure out why it's not working on this page. Below is the code I've been working with. Any help is much appreciated!</p>

<pre><code>from bs4 import BeautifulSoup
import urllib2

def make_soup(url):
    page = urllib2.urlopen(url)
    soupdata = BeautifulSoup(page, 'lxml')
    return soupdata

def get_player_totals():
    soup = make_soup("http://www.pro-football-reference.com/years/2015/")

    tableStats = soup.find('table', {'id':'team_stats'})

    return tableStats

print get_player_totals()
</code></pre>

<p>EDIT:</p>

<p>Thanks for all the help everyone. Both of the provided solutions below have been successful. Much appreciated!</p>

<p>A couple of thoughts here: first, as mentioned in the comments, the acutal table is commented out and is not <em>per se</em> part of the <code>DOM</code> (not accessible via a parser adhoc).<br>
In this situation, you can loop over the comments found and try to get the information via regular expressions (though this heavily discussed and mostly disliked on Stackoverflow, <a href="http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags"><strong>see here for more information</strong></a>). Last, but not least, I would recommend <a href="http://docs.python-requests.org/en/master/" rel="nofollow"><strong><code>requests</code></strong></a> rather than <code>urllib2</code>.  </p>

<p>That being said, here is a working code example:</p>

<pre><code>from bs4 import BeautifulSoup, Comment
import requests, re

def make_soup(url):
    r = requests.get(url)
    soupdata = BeautifulSoup(r.text, 'lxml')
    return soupdata

soup = make_soup("http://www.pro-football-reference.com/years/2015/")

# get the comments
comments = soup.findAll(text=lambda text:isinstance(text, Comment))

# look for table with the id "team_stats"
rx = re.compile(r'&lt;table.+?id="team_stats".+?&gt;[\s\S]+?&lt;/table&gt;')
for comment in comments:
    try:
        table = rx.search(comment.string).group(0)
        print(table)
        # break the loop if found
        break
    except:
        pass
</code></pre>

<p><strong>Always</strong> favour a parser solution over this one, especially the regex part is very error-prone.</p>

python web-scraping beautifulsoup bs4