Can't find all tables in website
<p>I'm trying to get the code for all the tables inside <a href="http://www.pro-football-reference.com/years/1932" rel="nofollow">http://www.pro-football-reference.com/years/1932</a>, but I'm only getting the first table.</p>

<p>I've tried switching the parser to lxml, but it still gives the length value of 1 for total tables.</p>

<p>It also doesn't give the all of the tags such as divs, tds ...etc</p>

<pre><code>from bs4 import BeautifulSoup
import requests

base_url = 'http://www.pro-football-reference.com/years/1932'

url = base_url

r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser')

print len(soup.find_all('table'))
</code></pre>

<p>This is because your page render the table by javascript.
So there is 2 ways to do that </p>

<ul>
<li>first one is using a scraping engine with javascript like Selenium.</li>
<li>second one is finding inside the html content and render it yourself.</li>
</ul>

<p>For this code I approach the 2nd one. Found that the table was hidden by <code>&lt;--</code> and <code>--&gt;</code>.
Just find all those things and replace it.</p>

<pre><code>import re
from bs4 import BeautifulSoup
import requests

base_url = 'http://www.pro-football-reference.com/years/1932/'

url = base_url

r = requests.get(url)
content = re.sub(r'(?m)^\&lt;!--.*\n?', '', r.content)
content = re.sub(r'(?m)^\--&gt;.*\n?', '', content)
soup = BeautifulSoup(content, 'html.parser')
print len(soup.find_all('table'))
for table in soup.find_all('table'):
    if table.findParent("table") is None:
        print "\n\n", str(table)
</code></pre>

python beautifulsoup