Web Scraping Javascript Using Python
<p>I am used to using BeautifulSoup to scrape a website, however this website is different. Upon soup.prettify() I get back Javascript code, lots of  stuff. I want to scrape this website for the data on the actual website (company name, telephone number etc). Is there a way of scraping these scripts such as Main.js to retrieve the data that is displayed on the website to me?  </p>

<p>Clear version:</p>

<p>Code is: </p>

<pre><code>&lt;script src="/docs/Main.js" type="text/javascript" language="javascript"&gt;&lt;/script&gt;
</code></pre>

<p>This holds the text that is on the website. I would like to scrape this text however it is populated using JS not HTML (which I used to use BeautifulSoup for).</p>

<p>You're asking if you can scrape text generated at runtime by Javascript. The answer is sort-of.</p>

<p>You'd need to run some kind of <a href="https://github.com/dhamaniasad/HeadlessBrowsers" rel="nofollow">headless browser</a>, like PhantomJS, in order to let the Javascript execute and populate the page. You'd then need to feed the HTML that the headless browser generates to BeautifulSoup in order to parse it.</p>

python python-2.7