beautiful soup - python - table scraping
<p>Trying to scrape a table from a website using beautiful soup in order to parse the data. How would I go about parsing it by its headers? So far, I can't even manage to print the entire table.Thanks in advance.</p>

<p>Here is the code:</p>

<pre><code>import urllib2
from bs4 import BeautifulSoup

optionstable = "http://www.barchart.com/options/optdailyvol?type=stocks"
page = urllib2.urlopen(optionstable)
soup = BeautifulSoup(page, 'lxml')

table = soup.find("div", {"class":"dataTables_wrapper","id": "options_wrapper"})

table1 = table.find_all('table')

print table1
</code></pre>

<p>The page is a dynamic page. It uses the <a href="https://datatables.net/" rel="nofollow">DataTables</a> jquery plugin to display the table. If you look at the page source you'll see an empty table. It is filled after the page is loaded.</p>

<p>You have two choices here. Either figure out which URL the Javascript code running in the browser accesses to get the table data, and hope that simply accessing this URL will work. If it does, the response will probably be in JSON, so you don't need to parse anything.</p>

<p>The second choice is to use a tool like Selenium or PhantomJS to load the page and run the Javascript code. Then you can access the already populated table and get its data.</p>

python parsing web-scraping beautifulsoup