How to scrape value from page that loads dynamicaly?
<p>The homepage of the website I'm trying to scrape displays four tabs, one of which reads "[Number] Available Jobs". I'm interested in scraping the [Number] value. When I inspect the page in Chrome, I can see the value enclosed within a <code>&lt;span&gt;</code> tag.</p>

<p><a href="http://i.stack.imgur.com/xeirT.png" rel="nofollow"><img src="http://i.stack.imgur.com/xeirT.png" alt="enter image description here"></a></p>

<p>However, there is nothing enclosed in that <code>&lt;span&gt;</code> tag when I view the page source directly. I was planning on using the Python <code>requests</code> module to make an HTTP GET request and then use regex to capture the value from the returned content. This is obviously not possible if the content doesn't contain the number I need.</p>

<p>My questions are:</p>

<ol>
<li><p>What is happening here? How can a value be dynamically loaded into a
page, displayed, and then not appear within the HTML source?</p></li>
<li><p>If the value doesn't appear in the page source, what can I do to
reach it?</p></li>
</ol>

<p>If the content doesn't appear in the page source then it is probably generated using javascript. For example the site might have a REST API that lists jobs, and the Javascript code could request the jobs from the API and use it to create the node in the DOM and attach it to the available jobs. That's just one possibility.</p>

<p>One way to scrap this information is to figure out how that javascript works and make your python scraper do the same thing (for example, if there is a simple REST API it is using, you just need to make a request to that same URL). Often that is not so easy, so another alternative is to do your scraping using a javascript capable browser like selenium.</p>

<p>One final thing I want to mention is <a href="http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454">that regular expressions are a fragile way to parse HTML</a>, you should generally prefer to use a library like BeautifulSoup.</p>

python html httprequest httpresponse