Table contents missing during extraction with selenium and phantomjs
<p>I am trying to extract data from the following table. However, the program returns an empty table with the "/table>". Notice that there are two classes with tbpopt, hence I am using "style" as an additional descriptor for the second one. The first tblopt shows up fine, the problem is with the second one. Thanks in advance.</p>

<pre><code>import time; import os; import os.path
import pandas as pd
from bs4 import BeautifulSoup, Tag
from selenium import webdriver
from selenium.webdriver.support.ui import Select

browser = webdriver.PhantomJS(r'C:\Users\abdurrub\Anaconda3\Scripts\phantomjs.exe')

browser.get('http://www.moneycontrol.com/stocks/fno/view_option_chain.php')

time.sleep(2)

soup = BeautifulSoup(browser.page_source, "lxml")
table = soup.select_one("table.tblopt")
print(soup.find('table', attrs={'class':'tblopt', 'style' :'width:100%;*width:100%'}))
</code></pre>

<p>My output is</p>

<pre><code>&lt;table border="0" cellpadding="0" cellspacing="0" class="tblopt" style="width:100%;*width:100%"&gt;
&lt;/table&gt;
</code></pre>

<p>If there are always two tables of <code>class='tblopt'</code> you could do something along the lines of:</p>

<pre><code>from selenium import webdriver
from bs4 import BeautifulSoup

if __name__ == '__main__':
    url = 'http://www.moneycontrol.com/stocks/fno/view_option_chain.php?sc_id=IRI&amp;sel_exp_date=2016-10-27'
    driver = webdriver.PhantomJS('&lt;yourPathToPhantomJS&gt;')
    driver.get(url)
    html = driver.page_source
    soup = BeautifulSoup(html)
    tbls = soup.find_all('table', {'class': 'tblopt'})
    print(tbls[0].get_text())
    print(tbls[1].get_text())
</code></pre>

python selenium web-scraping phantomjs