Trying to scrape this website using python but unable to get the required data
<p>I am trying to get the company names in this website <a href="https://siftery.com/microsoft-outlook" rel="nofollow">https://siftery.com/microsoft-outlook</a>
Basically it lists some companies that use Microsoft Outlook.
I used BeautifulSoup,requests,urllib and urllib2 but I still am not getting the names of the companies that use Microsoft Outlook not even in the first page of the website. </p>

<p>The code I wrote is below - </p>

<pre><code>r = requests.get('http://siftery.com/microsoft-outlook')

print(str(r.content))
f=open('abc.txt','w')
f.write(r.content)
f.close()
</code></pre>

<p>and part of the output that looks interesting is this -</p>

<p>({"name":"Marketing","handle":"marketing","categories":[{"name":"Marketing Automation","handle":"marketing-automation","external_id":"tgJ_49k7v4J-wV","parent_handle":null,"categories":[{"name":"Marketing Automation Platforms","handle":"marketing-automation-platforms","external_id":"tgJLE9aHoLdneT","parent_handle":"marketing-automation"},</p>

<p>BeautifulSoup also gives me the same output, so do the other libraries.
It seems like "external_id" is where the company name is ? I'm not sure. I also tried to manually find the name of a company for example Acxiom using gedit but couldn't find any occurrence.</p>

<p>That site loads the information using javascript that means that when you do the requests, the DOM is rendered without the information because it is loaded asynchronously, for sites like that you should use selenium.</p>

<p>Note:
Before you build a scraper you should look if the site has an api or end-points with CORS disabled, in your case you can get the information doing a post request to <code>https://siftery.com/product-json/&lt;product_name&gt;</code></p>

python web-scraping