Web scrapping table with hidden part using python
<p>IÂ´m trying to get the information from this table:</p>

<pre><code>&lt;table class="table4 table4-1 table4-1-1"&gt;&lt;thead&gt;&lt;tr&gt;&lt;th class="estilo1"&gt;No&lt;/th&gt;&lt;th class="estilo2"&gt;Si&lt;/th&gt;&lt;!--                                                        &lt;th&gt;&lt;div class="contenedor-vinculos6"&gt;&lt;a title="Ver mÃ¡s " class="vinculo-interrogacion" href="#"&gt;MÃ¡s informaciÃ³n&lt;/a&gt;&lt;/div&gt;&lt;/th&gt;--&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td class="estilo1"&gt;&lt;span class="estilo3"&gt;100%&lt;span class="numero-voto"&gt;(15)&lt;/span&gt;&lt;/span&gt;&lt;div class="grafica1 grafica1-desacuerdo"&gt;&lt;div class="item-grafica" style="width: 100%;"/&gt;&lt;/div&gt;&lt;/div&gt;&lt;/td&gt;&lt;td class="estilo2"&gt;&lt;span class="estilo3"&gt;0%&lt;span class="numero-voto"&gt;(0)&lt;/span&gt;&lt;/span&gt;&lt;div class="grafica1 grafica1-deacuerdo"&gt;&lt;div class="item-grafica" style="width: 0%;"/&gt;&lt;/div&gt;&lt;/div&gt;&lt;/td&gt;&lt;td&gt;&lt;span class="display-none"&gt;MÃ¡s informaciÃ³n&lt;/span&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;
</code></pre>

<p>IÂ´m doing the following in python3:</p>

<pre><code>req = Request('http://www.congresovisible.org/votaciones/10918/',headers=headers)
web_page = urlopen(req)
soup = BeautifulSoup(web_page.read(), 'html.parser')
table= soup.find_all('table', attrs={'class':'table4 table4-1 table4-1-1'})
</code></pre>

<p>This works but only shows part of the table, it excludes everything after: </p>

<pre><code>&lt;td class="estilo2"&gt;&lt;span class="estilo3...)
</code></pre>

<p>This is the output </p>

<pre><code>[&lt;table class="table4 table4-1 table4-1-1"&gt;&lt;thead&gt;&lt;tr&gt;&lt;th class="estilo1"&gt;No&lt;/th&gt;&lt;th class="estilo2"&gt;Si&lt;/th&gt;&lt;!--                                                        &lt;th&gt;&lt;div class="contenedor-vinculos6"&gt;&lt;a title="Ver mÃ¡s " class="vinculo-interrogacion" href="#"&gt;MÃ¡s informaciÃ³n&lt;/a&gt;&lt;/div&gt;&lt;/th&gt;--&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td class="estilo1"&gt;&lt;span class="estilo3"&gt;100%&lt;span class="numero-voto"&gt;(15)&lt;/span&gt;&lt;/span&gt;&lt;div class="grafica1 grafica1-desacuerdo"&gt;&lt;div class="item-grafica" style="width: 100%;"&gt;&lt;/div&gt;&lt;/div&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;]
</code></pre>

<p>How could I extract the whole table? </p>

<p>It is actually quite easy to solve. <code>html.parser</code> does not parse this kind of non-well-formed HTML well. Use a <em>more lenient</em> <code>html5lib</code> instead. This works for me:</p>

<pre><code>import requests
from bs4 import BeautifulSoup

response = requests.get('http://www.congresovisible.org/votaciones/10918/')
soup = BeautifulSoup(response.content, 'html5lib')
table = soup.find_all('table', attrs={'class':'table4 table4-1 table4-1-1'})
print(table)
</code></pre>

<p>Note that this requires <code>html5lib</code> package to be installed:</p>

<pre><code>pip install --upgrade html5lib
</code></pre>

<hr>

<p>By the way, <code>lxml</code> parser works as well:</p>

<pre><code>soup = BeautifulSoup(response.content, 'lxml')
</code></pre>

python web-scraping beautifulsoup html-table