BeautifulSoup webscraper issue: can't find certain divs/tables
<p>I'm having issues with scraping pro-football-reference.com. I'm trying to access the "Team Offense" table but can't seem to target the div/table.
The best I can do is:</p>

<pre><code>soup.find('div', {'id':'all_team_stats})
</code></pre>

<p>which doesn't return the table nor it's immediate div wrapper. The following attempts return "None":</p>

<pre><code>soup.find('div', {'id':'div_team_stats'})
soup.find('table', {'id':'team_stats'})
</code></pre>

<p>I've already scraped different pages simply by:</p>

<pre><code>soup.find('table', {'id':'table_id})
</code></pre>

<p>but I can't figure out why it's not working on this page. Below is the code I've been working with. Any help is much appreciated!</p>

<pre><code>from bs4 import BeautifulSoup
import urllib2

def make_soup(url):
    page = urllib2.urlopen(url)
    soupdata = BeautifulSoup(page, 'lxml')
    return soupdata

def get_player_totals():
    soup = make_soup("http://www.pro-football-reference.com/years/2015/")

    tableStats = soup.find('table', {'id':'team_stats'})

    return tableStats

print get_player_totals()
</code></pre>

<p>EDIT:</p>

<p>Thanks for all the help everyone. Both of the provided solutions below have been successful. Much appreciated!</p>

<p>Just remove the comments with <em>re.sub</em> before you pass to bs4:</p>

<pre><code>from bs4 import BeautifulSoup
import urllib2
import re
comm = re.compile("&lt;!--|--&gt;")
def make_soup(url):
    page = urllib2.urlopen(url)
    soupdata = BeautifulSoup(comm.sub("", page.read()), 'lxml')
    return soupdata

def get_player_totals():
    soup = make_soup("http://www.pro-football-reference.com/years/2015/")

    tableStats = soup.find('table', {'id':'team_stats'})

    return tableStats

print get_player_totals()
</code></pre>

<p>You will see the table when you run the code.</p>

python web-scraping beautifulsoup bs4