Missing class in beautifulsoup
<p>I am attempting to grab a div on the MTA info page. When I grab the html and parse it with BeautifulSoup it seems to be missing some data.</p>

<p>Here is my code so far</p>

<pre><code>from bs4 import BeautifulSoup
import urllib # access the web

# SUBWAY STATUS PROJECT
userURL = "http://www.mta.info" # MTA SITE

htmlfile = urllib.urlopen(userURL) #creates html file
htmldoc = htmlfile.read()   #creates html text

soup = BeautifulSoup(htmldoc, 'html.parser')    

subChart = soup.find( id = 'subwayDiv')

print subChart
</code></pre>

<p>I am using print just to be sure that I am getting all of the data. I see that I am missing some information that I am trying to grab. If I look to the page myself I can see that I am missing a div with the class that shows the subway status.</p>

<p>I am very new to programming so please mind my ignorance</p>

<p>The data is retrieved with an ajax request, you can get the info in <em>json</em> format, the only thing you need to pass it a <em>timestamp</em> which you can get with <em>time.time()</em> then just parse it with the  <a href="https://docs.python.org/2/library/json.html" rel="nofollow">json</a> library:</p>

<pre><code>from time import time
from json import load, loads
import urllib

url  = "http://www.mta.info/service_status_json/{}".format(int(time()))

json_dict = loads(load(urllib.urlopen(url))) 

from pprint import pprint as pp
pp(json_dict)
</code></pre>

<p>I won't add all the output as there is too much but using the <code>"BT"</code> we get:</p>

<pre><code>{u'line': [{u'Date': {},
            u'Time': {},
            u'name': u'Bronx-Whitestone',
            u'status': u'GOOD SERVICE',
            u'text': {}},
           {u'Date': {},
            u'Time': {},
            u'name': u'Cross Bay',
            u'status': u'GOOD SERVICE',
            u'text': {}},
           {u'Date': {},
            u'Time': {},
            u'name': u'Henry Hudson',
            u'status': u'GOOD SERVICE',
            u'text': {}},
           {u'Date': u'09/16/2016',
            u'Time': u' 5:57AM',
            u'name': u'Hugh L. Carey',
            u'status': u'SERVICE CHANGE',
            u'text': u"                    &lt;span class='TitleServiceChange' &gt;Service Change&lt;/span&gt;                    &lt;span class='DateStyle'&gt;                    &amp;nbsp;Posted:&amp;nbsp;09/16/2016&amp;nbsp; 5:57AM                    &lt;/span&gt;&lt;br/&gt;&lt;br/&gt;                  HLC - HOV Lane Open 6 AM to 10 AM. Two-Way Operations in effect. Three (3) lanes Manhattan-bound. One (1) lane Brooklyn-bound.                &lt;br/&gt;&lt;br/&gt;              "},
           {u'Date': {},
            u'Time': {},
            u'name': u'Marine Parkway',
            u'status': u'GOOD SERVICE',
            u'text': {}},
           {u'Date': u'09/16/2016',
            u'Time': u' 5:57AM',
            u'name': u'Queens Midtown',
            u'status': u'SERVICE CHANGE',
            u'text': u"                    &lt;span class='TitleServiceChange' &gt;Service Change&lt;/span&gt;                    &lt;span class='DateStyle'&gt;                    &amp;nbsp;Posted:&amp;nbsp;09/16/2016&amp;nbsp; 5:57AM                    &lt;/span&gt;&lt;br/&gt;&lt;br/&gt;                  QMT - HOV Lane Open 6 AM to 10 AM. Two-Way Operation in effect. Three (3) lanes Manhattan bound. One (1) lane Queens bound.                &lt;br/&gt;&lt;br/&gt;                                  &lt;span class='TitlePlannedWork' &gt;Planned Work&lt;/span&gt;                    &lt;br/&gt;                  &lt;P  style='MARGIN: 0in 0in 0pt'&gt;&lt;SPAN style=''Times New Roman';2016; Queens-Midtown Tunnel downtown exit; One lane closed. Use 37&lt;SUP&gt;th&lt;/SUP&gt;&lt;/FONT&gt;&lt;FONT size=3&gt; St tunnel exit for access to 2&lt;/FONT&gt;&lt;SUP&gt;&lt;FONT size=3&gt;nd&lt;/FONT&gt;&lt;/SUP&gt;&lt;FONT size=3&gt; Ave. Motorists should allow extra time and may wish to use an alternate route if possible' Drivers should expect delays and plan accordingly. Motorists can sign up for MTA e-mail or text alerts at &lt;/FONT&gt;&lt;SPAN style='COLOR: blue'&gt;&lt;A href='http://www.mta.info/'&gt;&lt;SPAN style='COLOR: #0563c1'&gt;&lt;FONT size=3&gt;www.mta.info&lt;/FONT&gt;&lt;/SPAN&gt;&lt;/A&gt;&lt;FONT size=3&gt; &lt;/FONT&gt;&lt;/SPAN&gt;&lt;FONT size=3&gt;and check the Bridges and Tunnels homepage or Facebook page for the latest information on this planned work.&lt;/FONT&gt;&lt;/FONT&gt;&lt;/SPAN&gt;&lt;/P&gt;                &lt;br/&gt;&lt;br/&gt;                                  &lt;span class='TitlePlannedWork' &gt;Planned Work&lt;/span&gt;                    &lt;br/&gt;                  QMT- MANHATTAN PLAZA WORK REQUIRES CLOSURE OF 'CROSSTOWN' LANES FOR 2 MONTHS. CUSTOMERS SEEKING A CROSSTOWN MANHATTAN ROUTE USE THE UPTOWN LANES; EXPECT DELAYS.                &lt;br/&gt;&lt;br/&gt;              "},
           {u'Date': u'08/15/2016',
            u'Time': u' 3:56PM',
            u'name': u'Robert F. Kennedy',
            u'status': u'PLANNED WORK',
            u'text': u"                    &lt;span class='TitlePlannedWork' &gt;Planned Work&lt;/span&gt;                    &lt;br/&gt;                  &lt;P  style='MARGIN: 0in 0in 0pt'&gt;&lt;SPAN style='COLOR: #1f497d'&gt;&lt;FONT size=3 face=Calibri&gt;Starting Monday, August 15, 2016 and through early 2018, one lane will be closed on the Queens-to-Manhattan ramp at the Robert F. Kennedy Bridge for roadway rehabilitation. In addition, overnight on Thursday, August 18 and Friday, August 19, there will be a series of intermittent FULL ramp closures, lasting 15-20 minutes each.&lt;/FONT&gt;&lt;/SPAN&gt;&lt;/P&gt;                &lt;br/&gt;&lt;br/&gt;              "},
           {u'Date': {},
            u'Time': {},
            u'name': u'Throgs Neck',
            u'status': u'GOOD SERVICE',
            u'text': {}},
           {u'Date': u'09/16/2016',
            u'Time': u' 5:28AM',
            u'name': u'Verrazano-Narrows',
            u'status': u'PLANNED WORK',
            u'text': u"                    &lt;span class='TitlePlannedWork' &gt;Planned Work&lt;/span&gt;                    &lt;br/&gt;                  VNB: PLANNED WORK; S. I. BOUND LOWER LEVEL - ONE LANE CLOSED; EXPECT DELAYS.                &lt;br/&gt;&lt;br/&gt;              "}]}
</code></pre>

<p>So you just need to go through the dict and pick out what you want.</p>

python html parsing beautifulsoup