Using BeautifulSoup to Search Through Yahoo Finance
<p>I'm trying to pull information from the 'Key Statistics' page for a ticker in Yahoo (since this isn't supported in the Pandas library). </p>

<p>Example for AAPL:</p>

<pre><code>from bs4 import BeautifulSoup
import requests

url = 'http://finance.yahoo.com/quote/AAPL/key-statistics?p=AAPL'
page = requests.get(url)
soup = BeautifulSoup(page.text, 'lxml')

enterpriseValue = soup.findAll('$ENTERPRISE_VALUE', attrs={'class': 'yfnc_tablehead1'}) #HTML tag for where enterprise value is located

print(enterpriseValue)
</code></pre>

<p>Edit: thanks Andy!</p>

<p>Question: This is printing an empty array. How do I change my <code>findAll</code> to return <code>598.56B</code>?</p>

<p>Well, the reason the list that <code>find_all</code> returns is empty is because that data is generated with a separate call that isn't completed by just sending a <code>GET</code> request to that URL.  If you look through the Network tab on Chrome/Firefox and filter by XHR, by examining the requests and responses of each network action, you can find what you URL you ought to be sending the <code>GET</code> request too.  </p>

<p>In this case, it's <code>https://query2.finance.yahoo.com/v10/finance/quoteSummary/AAPL?formatted=true&amp;crumb=8ldhetOu7RJ&amp;lang=en-US&amp;region=US&amp;modules=defaultKeyStatistics%2CfinancialData%2CcalendarEvents&amp;corsDomain=finance.yahoo.com</code>, as we can see here:
<a href="http://i.stack.imgur.com/B4TBz.png" rel="nofollow"><img src="http://i.stack.imgur.com/B4TBz.png" alt="enter image description here"></a></p>

<p>So, how do we recreate this? Simple! :</p>

<pre><code>from bs4 import BeautifulSoup
import requests

r = requests.get('https://query2.finance.yahoo.com/v10/finance/quoteSummary/AAPL?formatted=true&amp;crumb=8ldhetOu7RJ&amp;lang=en-US&amp;region=US&amp;modules=defaultKeyStatistics%2CfinancialData%2CcalendarEvents&amp;corsDomain=finance.yahoo.com')
data = r.json()
</code></pre>

<p>This will return the <code>JSON</code> response as a <code>dict</code>.  From there, navigate through the <code>dict</code> until you find the data you're after:</p>

<pre><code>financial_data = data['quoteSummary']['result'][0]['defaultKeyStatistics']
enterprise_value_dict = financial_data['enterpriseValue']
print(enterprise_value_dict)
&gt;&gt;&gt; {'fmt': '598.56B', 'raw': 598563094528, 'longFmt': '598,563,094,528'}
print(enterprise_value_dict['fmt'])
&gt;&gt;&gt; '598.56B'
</code></pre>

python beautifulsoup yahoo-finance