BeautifulSoup parse table data that doesn't load immediately
<p>I'm trying to download earnings announcement data from <a href="https://www.zacks.com/stock/research/MMM/earnings-announcements" rel="nofollow">https://www.zacks.com/stock/research/MMM/earnings-announcements</a> using beautifulsoup.  When I look at the tables, the table I am interested in (earnings_announcements_earnings_table) shows only "Loading Dataâ¦".  However, if I print the entire contents of the soup, I do see the information I am looking for is in there.  I can isolate this data just a "script" element, but that contains a lot of other unwanted info.  How can I select and parse just the specific data I'm looking for, which is the contents of the "earnings_announcements_earnings_table" table, which looks like this:</p>

<p>"    var obj = {
              "earnings_announcements_earnings_table"   : 
             [  [ "10/25/2016", "9/2016", "$2.14", "--", "--", "Before Open" ] ,..."</p>

<p>Here is what I have so far:</p>

<pre><code>from urllib import request
from urllib import error
from bs4 import BeautifulSoup

def download_parse_earnings(symbol):

request_string = "https://www.zacks.com/stock/research/%s/earnings-announcements" % symbol
print(request_string)

try:
    web = request.urlopen(request_string)
except error.HTTPError:
    return

soup = BeautifulSoup(web.read(), 'lxml')
data = soup.find_all("script")[28].string
print(data)
</code></pre>

<p>One way would be to fire up Selenium and make use of its Javascript engine. That's not really simple, and I offer here an alternative solution which is a total hack, but it should work for the pages you are interested in.</p>

<p>Assuming the pages are generated automatically, we observe that the data you want is in (continuing from you program):</p>

<pre><code>import json
earnings = json.loads(data.split('var obj =')[1].splitlines()[2])
</code></pre>

<p>This is leveraging the fact that Javascript objects are JSON, and so we read directly from the source. The result is a list of lists like this one:</p>

<pre><code>[['10/25/2016', '9/2016', '.14', '--', '--', 'Before Open'],
 ['7/26/2016',
  '6/2016',
  '.08',
  '.08',
  '&lt;div class=right pos_na showinline&gt;0.00 (0.00%)&lt;/div&gt;',
  'Before Open'],
 ['4/26/2016',
  '3/2016',
  '.92',
  '.05',
  '&lt;div class=right pos positive pos_icon showinline up&gt;0.13 (6.77%)&lt;/div&gt;',
  'Before Open'],
 ['1/26/2016',
  '12/2015',
  '.62',
  '.80',
  '&lt;div class=right pos positive pos_icon showinline up&gt;0.18 (11.11%)&lt;/div&gt;',
  'Before Open'],
 ['10/22/2015',
  '9/2015',
  '.01',
  '.05',
  '&lt;div class=right pos positive pos_icon showinline up&gt;0.04 (1.99%)&lt;/div&gt;',
  'Before Open'],
...
]
</code></pre>

<p>The first element corresponds to the first row of the table, i.e. the header. You just have to clean up the data now.</p>

python beautifulsoup