BeautifulSoup parse table data that doesn't load immediately
<p>I'm trying to download earnings announcement data from <a href="https://www.zacks.com/stock/research/MMM/earnings-announcements" rel="nofollow">https://www.zacks.com/stock/research/MMM/earnings-announcements</a> using beautifulsoup.  When I look at the tables, the table I am interested in (earnings_announcements_earnings_table) shows only "Loading Dataâ¦".  However, if I print the entire contents of the soup, I do see the information I am looking for is in there.  I can isolate this data just a "script" element, but that contains a lot of other unwanted info.  How can I select and parse just the specific data I'm looking for, which is the contents of the "earnings_announcements_earnings_table" table, which looks like this:</p>

<p>"    var obj = {
              "earnings_announcements_earnings_table"   : 
             [  [ "10/25/2016", "9/2016", "$2.14", "--", "--", "Before Open" ] ,..."</p>

<p>Here is what I have so far:</p>

<pre><code>from urllib import request
from urllib import error
from bs4 import BeautifulSoup

def download_parse_earnings(symbol):

request_string = "https://www.zacks.com/stock/research/%s/earnings-announcements" % symbol
print(request_string)

try:
    web = request.urlopen(request_string)
except error.HTTPError:
    return

soup = BeautifulSoup(web.read(), 'lxml')
data = soup.find_all("script")[28].string
print(data)
</code></pre>

<p>Without using Selenium but still using json as in the first answer you can dig out the content you need with BS.</p>

<pre><code>&gt;&gt;&gt; from bs4 import BeautifulSoup
&gt;&gt;&gt; from urllib import request
&gt;&gt;&gt; URL='https://www.zacks.com/stock/research/MMM/earnings-announcements'
&gt;&gt;&gt; HTML=request.urlopen(URL).read()
&gt;&gt;&gt; soup=BeautifulSoup(HTML)
&gt;&gt;&gt; import json
&gt;&gt;&gt; scripts=soup.findAll('script')
&gt;&gt;&gt; len(scripts)
36

&gt;&gt;&gt; for script in scripts:
...     if script.has_attr('type') and script.attrs['type']=='text/javascript' and script.text.strip().startswith('$(document).ready(function()'):
...         break
</code></pre>

<p>With this the javascript becomes available as script.text. You would still need to do something mildly clever to extract the lines shown in <a href="http://stackoverflow.com/users/448496/rubik">Rubik's</a> answer. Nothing like impossible though.</p>

python beautifulsoup