Python BeautifulSoup scrape Yahoo Finance value
<p>I am attempting to scrape the 'Full Time Employees' value of 110,000 from the Yahoo finance website. </p>

<p>The URL is: <code>http://finance.yahoo.com/quote/AAPL/profile?p=AAPL</code></p>

<p>I have tried using Beautiful soup, but I can't find the value on the page.  When I look in the DOM explorer in IE, I can see it.  It has a  tag with a parent tag  which has a parent <p> which has a parent .  The actual value is in a custom class of <code>data-react-id</code>.</p>

<p>code I have tried:</p>

<pre><code>from bs4 import BeautifulSoup as bs
html=`http://finance.yahoo.com/quote/AAPL/profile?p=AAPL`
r = requests.get(html).content
soup = bs(r)
</code></pre>

<p>Not sure where to go.</p>

<p>The problem is in the "requests" related part - <em>the page you download with <code>requests</code> is not the same as you see in the browser</em>. Browser executed all of the javascript, made multiple asynchronous requests needed to load this page. And, this particular page is quite <em>dynamic</em> itself. There is a lot happening on the "client-side".</p>

<p>What you can do is to load this page in a real browser automated by <a href="http://selenium-python.readthedocs.io/" rel="nofollow"><code>selenium</code></a>. Working example:</p>

<pre><code>from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait

driver = webdriver.Chrome()
driver.maximize_window()
driver.get("http://finance.yahoo.com/quote/AAPL/profile?p=AAPL")

# wait for the Full Time Employees to be visible
wait = WebDriverWait(driver, 10)
employees = wait.until(EC.visibility_of_element_located((By.XPATH, "//span[. = 'Full Time Employees']/following-sibling::strong")))
print(employees.text)

driver.close()
</code></pre>

<p>Prints <code>110,000</code>.</p>

python beautifulsoup