How to understand this raw HTML of Yahoo! Finance when retrieving data using Python?
<p>I've been trying to retrieve stock price from Yahoo! Finance, like for <a href="http://finance.yahoo.com/quote/AAPL/profile?p=AAPL" rel="nofollow">Apple Inc.</a>. My code is like this:(using Python 2)</p>

<pre><code>import requests
from bs4 import BeautifulSoup as bs

html='http://finance.yahoo.com/quote/AAPL/profile?p=AAPL'
r = requests.get(html)
soup = bs(r.text)
</code></pre>

<p>The problem is when I see raw HTML behind this webpage, the class is dynamic, see figure below. This makes it hard for BeautifulSoup to get tags. How to understand the class and how to get data?</p>

<p><a href="http://i.stack.imgur.com/bITK0.png" rel="nofollow">HTML of Yahoo! Finance page</a></p>

<p>PS: 1) I know pandas_datareader.data, but that's for historical data. I want the real-time stock data;</p>

<p>2) I don't want to use selenium to open a new browser window.</p>

<p>Not sure what you mean by 'dynamic' in this case, but have you considered using CSS selectors?</p>

<p>With Beautifulsoup you could get it e.g like this:</p>

<pre><code>soup.select('div#quote-header-info section span')[0]  
</code></pre>

<p>And there are some variations you could use on the pattern, such as using the '>' filter.</p>

<p>You could get the same with just <code>lxml</code>, no need for BeautifulSoup:</p>

<pre><code>import lxml.html as html
page = html.parse(url).getroot()
content = page.cssselect('div#quote-header-info section &gt; span:first-child')[0].text
</code></pre>

<p>Which immediately illustrates a more specific selector. </p>

<p>If you're interested in more efficient DOM-traversal, research xpaths.</p>

python html beautifulsoup web-crawler yahoo-finance