Can't extract data from beautiful_soup object
<p>I'm crawling a website(<a href="https://www.zhihu.com/people/xie-ke-41/followers" rel="nofollow">https://www.zhihu.com/people/xie-ke-41/followers</a>), and i want to get all the followers' information. As you can see, some followers' information is bring with AJAX, I use the developers' tool in chrome and locate the url  <a href="http://%22https://www.zhihu.com/node/ProfileFollowersListV2%22" rel="nofollow">the url which has followers' information</a></p>

<p>my code:</p>

<pre><code>import requests
from bs4 import BeautifulSoup


zhihu_rl = 'https://www.zhihu.com/node/ProfileFollowersListV2'

data = {
'method': 'next',
'params': '{"offset":20,"order_by":"created","hash_id":"86858a7a4aa77d290364625efcaacb70"}'}

headers = {
'Host': 'www.zhihu.com',
'Origin': 'https://www.zhihu.com',
'Referer': 'https://www.zhihu.com/people/xie-ke-41/followers',
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36',
'X-Requested-With': 'XMLHttpRequest',
'X-Xsrftoken': 'foo',
'Cookie':'xxxxxxxxxxxx'}

rep = requests.post(url=zhihu_rl, data=data, headers=headers)

bsobj = BeautifulSoup(rep.text,  'html.parser')

print(bsobj.find_all('div', {'class': "zm-profile-card zm-profile-section-item zg-clear no-hovercard"}))
</code></pre>

<p>and an empty list returned.
i can see the information is developer's tool:
<a href="http://i.stack.imgur.com/PetCj.png" rel="nofollow"><img src="http://i.stack.imgur.com/PetCj.png" alt="thr information i see in developers&#39; tool"></a>
,why can't bs4 extract them?
PS:i can get all the div,but when i limit the attributes.it failed</p>

<p>You have used good header combination otherwise the server may not recognize your header and it thinks that you don't have javascript enabled. In limiting the attribute use . for class and # for id. Other CSS selectors will also work fine. You need also to use <a href="http://www.seleniumhq.org/" rel="nofollow">Selenium</a> for the javascript execution(ajax calls) since beautifulsoup lack this feature
Finally, make sure the website doesn't have anti-scraping protection. In that case, you need to use javascript runtimes like <a href="https://github.com/PiotrDabkowski/Js2Py" rel="nofollow">Js2Py</a></p>

python beautifulsoup web-crawler