Can't extract data from beautiful_soup object
<p>I'm crawling a website(<a href="https://www.zhihu.com/people/xie-ke-41/followers" rel="nofollow">https://www.zhihu.com/people/xie-ke-41/followers</a>), and i want to get all the followers' information. As you can see, some followers' information is bring with AJAX, I use the developers' tool in chrome and locate the url  <a href="http://%22https://www.zhihu.com/node/ProfileFollowersListV2%22" rel="nofollow">the url which has followers' information</a></p>

<p>my code:</p>

<pre><code>import requests
from bs4 import BeautifulSoup


zhihu_rl = 'https://www.zhihu.com/node/ProfileFollowersListV2'

data = {
'method': 'next',
'params': '{"offset":20,"order_by":"created","hash_id":"86858a7a4aa77d290364625efcaacb70"}'}

headers = {
'Host': 'www.zhihu.com',
'Origin': 'https://www.zhihu.com',
'Referer': 'https://www.zhihu.com/people/xie-ke-41/followers',
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36',
'X-Requested-With': 'XMLHttpRequest',
'X-Xsrftoken': 'foo',
'Cookie':'xxxxxxxxxxxx'}

rep = requests.post(url=zhihu_rl, data=data, headers=headers)

bsobj = BeautifulSoup(rep.text,  'html.parser')

print(bsobj.find_all('div', {'class': "zm-profile-card zm-profile-section-item zg-clear no-hovercard"}))
</code></pre>

<p>and an empty list returned.
i can see the information is developer's tool:
<a href="http://i.stack.imgur.com/PetCj.png" rel="nofollow"><img src="http://i.stack.imgur.com/PetCj.png" alt="thr information i see in developers&#39; tool"></a>
,why can't bs4 extract them?
PS:i can get all the div,but when i limit the attributes.it failed</p>

<p>The problem is you have escaped json, if you print the bsobj you can see output like:</p>

<pre><code>{"r":0,
 "msg": ["&lt;div class=\"zm-profile-card zm-profile-section-item zg-clear no-hovercard\"&gt;\n&lt;div class=\"zg-right\"&gt;\n&lt;button data-follow=\"m:button\" data-id=\"6327483c9e474097e7dbb2493a7f277c\" class=\"zg-btn zg-btn-follow zm-rich-follow-btn small nth-0\"&gt;\u5173\u6ce8\u4ed6&lt;\/button&gt;\n&lt;\/div&gt;\n&lt;a title=\"\u738b\u5728\u9014\"\ndata-hovercard=\"p$t$wang-zai-tu-81\"\nclass=\"zm-item-link-avatar\"\nhref=\"\/people\/wang-zai-tu-81\"&gt;\n&lt;img src=\"https:\/\/pic1.zhimg.com\/da8e974dc_m.jpg\" class=\"zm-item-img-avatar\"&gt;\n&lt;\/a&gt;\n&lt;div class=\"zm-list-content-medium\"&gt;\n&lt;h2 class=\"zm-list-content-title\"&gt;&lt;span class=\"author-link-line\"&gt;\n&lt;a data-hovercard=\"p$t$wang-zai-tu-81\" href=\"https:\/\/www.zhihu.com\/people\/wang-zai-tu-81\" class=\"zg-link author-link\" title=\"\u738b\u5728\u9014\"\n&gt;\u738b\u5728\u9014&lt;\/a&gt;&lt;\/span&gt;&lt;\/h2&gt;\n\n&lt;div class=\"summary-wrapper summary-wrapper--medium\"&gt;\n\n&lt;span class=\"bio\"&gt;&lt;\/span&gt;\n&lt;\/div&gt;\n&lt;div class=\"details zg-gray\"&gt;\n&lt;a target=\"_blank\" href=\"\/people\/wang-zai-tu-81\/followers\" class=\"zg-link-gray-normal\"&gt;1 \u5173\u6ce8\u8005&lt;\/a&gt;\n\/\n&lt;a target=\"_blank\" href=\"\/people\/wang-zai-tu-81\/asks\" class=\"zg-link-gray-normal\"&gt;0 \u63d0\u95ee&lt;\/a&gt;\n\/\n&lt;a target=\"_blank\" href=\"\/people\/wang-zai-tu-81\/answers\" class=\"zg-link-gray-normal\"&gt;1 \u56de\u7b54&lt;\/a&gt;\n\/\n&lt;a target=\"_blank\" href=\"\/people\/wang-zai-tu-81\" class=\"zg-link-gray-normal\"&gt;0 \u8d5e\u540c&lt;\/a&gt;\n&lt;\/div&gt;\n\n&lt;\/div&gt;\n&lt;\/div&gt;","&lt;div class=\"zm-profile-card zm-profile-section-item zg-clear no-hovercard\"&gt;\n&lt;div class=\"zg-right\"&gt;\n&lt;button data-follow=\"m:button\" data-id=\"a3596eaecae6f05f0ddf95dfcc6b5517\" class=\"zg-btn zg-btn-follow zm-rich-follow-btn small nth-0\"&gt;\u5173\u6ce8&lt;\/button&gt;\n&lt;\/div&gt;\n&lt;a title=\"\u7075\u9b42\"\ndata-hovercard=\"p$t$ling-hun-30-21\"\nclass=\"zm-item-link-avatar\"\nhref=\"\/people\/ling-hun-30-21\"&gt;\n&lt;img src=\"https:\/\/pic1.zhimg.com\/da8e974dc_m.jpg\" class=\"zm-item-img-avatar\"&gt;\n&lt;\/a&gt;\n&lt;div class=\"zm-list-content-medium\"&gt;\n&lt;h2 class=\"zm-list-content-title\"&gt;&lt;span class=\"author-link-line\"&gt;\n&lt;a data-hovercard=\"p$t$ling-hun-30-21\" href=\"https:\/\/www.zhihu.com\/people\/ling-hun-30-21\" class=\"zg-link author-link\" title=\"\u7075\u9b42\"\n&gt;\u7075\u9b42&lt;\/a&gt;&lt;\/span&gt;&lt;\/h2&gt;\n\n&lt;div class=\"summary-wrapper summary-wrapper--medium\"&gt;\n\n&lt;span class=\"bio\"&gt;&lt;\/span&gt;\n&lt;\/div&gt;\n&lt;div class=\"details zg-gray\"&gt;\n&lt;a target=\"_blank\" href=\"\/people\/ling-hun-30-21\/followers\" class=\"zg-link-gray-normal\"&gt;0 \u5173\u6ce8\u8005&lt;\/a&gt;\n\/\n&lt;a target=\"_blank\" href=\"\/people\/ling-hun-30-21\/asks\" class=\"zg-link-gray-normal\"&gt;0 \u63d0\u95ee&lt;\/a&gt;\n\/\n&lt;a target=\"_blank\" href=\"\/people\/ling-hun-30-21\/answers\" class=\"zg-link-gray-normal\"&gt;0 \u56de\u7b54&lt;\/a&gt;\n\/\n&lt;a target=\"_blank\" href=\"\/people\/ling-hun-30-21\" class=\"zg-link-gray-normal\"&gt;0 \u8d5e\u540c&lt;\/a&gt;\n&lt;\/div&gt;\n\n&lt;\/div&gt;\n&lt;\/div&gt;","&lt;div class=\"zm-profile-card zm-profile-section-item zg-clear no-hovercard\"&gt;\n&lt;div class=\"zg-right\"&gt;\n&lt;button data-follow=\"m:button\" data-id=\"74fad3af2b93f7da69c37eda64c31037\" class=\"zg-btn zg-btn-follow zm-rich-follow-btn small nth-0\"&gt;\u5173\u6ce8&lt;\/button&gt;\n&lt;\/div&gt;\n&lt;a title=\"\u5f90\u6668\"\ndata-hovercard=\"p$t$xu-chen-77-49\"\nclass=\"zm-item-link-avatar\"\nhref=\"\/people\/xu-chen-77-49\"&gt;\n&lt;img src=\"https:\/\/pic1.zhimg.com\/da8e974dc_m.jpg\" class=\"zm-item-img-avatar\"&gt;\n&lt;\/a&gt;\n&lt;div class=\"zm-list-content-medium\"&gt;\n&lt;h2 class=\"zm-list-content-title\"&gt;&lt;span class=\"author-link-line\"&gt;\n&lt;a data-hovercard=\"p$t$xu-chen-77-49\" href=\"https:\/\/www.zhihu.com\/people\/xu-chen-77-49\" class=\"zg-link author-link\" title=\"\u5f90\u6668\"\n&gt;\u5f90\u6668&lt;\/a&gt;&lt;\/span&gt;&lt;\/h2&gt;\n\n&lt;div class=\"summary-wrapper summary-wrapper--medium\"&gt;\n\n&lt;span class=\"bio\"&gt;\u4f1a\u8ba1\u5e08&lt;\/span&gt;\n&lt;\/div&gt;\n&lt;div class=\"details zg-gray\"&gt;\n&lt;a target=\"_blank\" href=\"\/people\/xu-chen-77-49\/followers\" class=\"zg-link-gray-normal\"&gt;0 \u5173\u6ce8\u8005&lt;\/a&gt;\n\/\n&lt;a target=\"_blank\" href=\"\/people\/xu-chen-77-49\/asks\" class=\"zg-link-gray-normal\"&gt;0 \u63d0\u95ee&lt;\/a&gt;\n\/\n&lt;a target=\"_blank\" href=\"\/people\/xu-chen-77-49\/answers\" class=\"zg-link-gray-normal\"&gt;0 \u56de\u7b54&lt;\/a&gt;\n\/\n&lt;a target=\"_blank\" href=\"\/people\/xu-chen-77-49\" class=\"zg-link-gray-normal\"&gt;0 \u8d5e\u540c&lt;\/a&gt;\n&lt;\/div&gt;\n\n&lt;\/div&gt;\n&lt;\/div&gt;","&lt;div class=\"zm-profile-card zm-profile-section-item zg-clear no-hovercard\"&gt;\n&lt;div class=\"zg-right\"&gt;\n&lt;button data-follow=\"m:button\" data-id=\"032b36abfbe05a30913c794a4b099629\" class=\"zg-btn zg-btn-follow zm-rich-follow-btn small nth-0\"&gt;\u5173\u6ce8\u5979&lt;\/button&gt;\n&lt;\/div&gt;\n&lt;a title=\"Shuai Zhang\"\ndata-hovercard=\"p$t$shuai-zhang-49\"\nclass=\"zm-item-link-avatar\"\nhref=\"\/people\/shuai-zhang-49\"&gt;\n&lt;img src=\"https:\/\/pic2.zhimg.com\/v2-8aa42ff00873460e29444d62ff51acfd_m.jpg\" class=\"zm-item-img-avatar\"&gt;\n&lt;\/a&gt;\n&lt;div class=\"zm-list-content-medium\"&gt;\n&lt;h2 class=\"zm-list-content-title\"&gt;&lt;span class=\"author-link-line\"&gt;\n&lt;a data-hovercard=\"p$t$shuai-zhang-49\" href=\"https:\/\/www.zhihu.com\/people\/shuai-zhang-49\" class=\"zg-link author-link\" title=\"Shuai Zhang\"\n&gt;Shuai Zhang&lt;\/a&gt;&lt;\/span&gt;&lt;\/h2&gt;\n\n&lt;div class=\"summary-wrapper summary-wrapper--medium\"&gt;\n\n&lt;span class=\"bio\"&gt;&lt;\/span&gt;\n&lt;\/div&gt;\n&lt;div class=\"details zg-gray\"&gt;\n&lt;a target=\"_blank\" href=\"\/people\/shuai-zhang-49\/followers\" class=\"zg-link-gray-normal\"&gt;79 \u5173\u6ce8\u8005&lt;\/a&gt;\n\/\n&lt;a target=\"_blank\" href=\"\/people\/shuai-zhang-49\/asks\" class=\"zg-link-gray-normal\"&gt;1 \u63d0\u95ee&lt;\/a&gt;\n\/\n&lt;a target=\"_blank\" href=\"\/people\/shuai-zhang-49\/answers\" class=\"zg-link-gray-normal\"&gt;119 \u56de\u7b54&lt;\/a&gt;\n\/\n&lt;a target=\"_blank\" href=\"\/people\/shuai-zhang-49\" class=\"zg-link-gray-normal\"&gt;174 \u8d5e\u540c&lt;\/a&gt;\n&lt;\/div&gt;\n\n&lt;\/div&gt;\n&lt;\/div&gt;","&lt;div class=\"zm-profile-card zm-profile-section-item zg-clear no-hovercard\"&gt;\n&lt;div class=\"zg-right\"&gt;\n&lt;button data-follow=\"m:button\" data-id=\"6388162f5357ca1bd872dc0b6efe4802\" class=\"zg-btn zg-btn-follow zm-rich-follow-btn small nth-0\"&gt;\u5173\u6ce8\u4ed6&lt;\/button&gt;\n&lt;\/div&gt;\n&lt;a title=\"\u5468\u5468\"\ndata-hovercard=\"p$t$zhou-zhou-69-22\"\nclass=\"zm-item-link-avatar\"\nhref=\"\/people\/zhou-zhou-69-22\"&gt;\n&lt;img src=\"https:\/\/pic1.zhimg.com\/da8e974dc_m.jpg\" class=\"zm-item-img-avatar\"&gt;\n&lt;\/a&gt;\n&lt;div class=\"zm-list-content-medium\"&gt;\n&lt;h2 class=\"zm-list-content-title\"&gt;&lt;span class=\"author-link-line\"&gt;\n&lt;a data-hovercard=\"p$t$zhou-zhou-69-22\" href=\"https:\/\/www.zhihu.com\/people\/zhou-zhou-69-22\" class=\"zg-link author-link\" title=\"\u5468\u5468\"\n&gt;\u5468\u5468&lt;\/a&gt;&lt;\/span&gt;&lt;\/h2&gt;\n\n&lt;div class=\"summary-wrapper summary-wrapper--medium\"&gt;\n\n&lt;span class=\"bio\"&gt;&lt;\/span&gt;\n&lt;\/div&gt;\n&lt;div class=\"details zg-gray\"&gt;\n&lt;a target=\"_blank\" href=\"\/people\/zhou-zhou-69-22\/followers\" class=\"zg-link-gray-normal\"&gt;4 \u5173\u6ce8\u8005&lt;\/a&gt;\n\/\n&lt;a target=\"_blank\" href=\"\/people\/zhou-zhou-69-22\/asks\" class=\"zg-link-gray-normal\"&gt;0 \u63d0\u95ee&lt;\/a&gt;\n\/\n&lt;a target=\"_blank\" href=\"\/people\/zhou-zhou-69-22\/answers\" class=\"zg-link-gray-normal\"&gt;7 \u56de\u7b54&lt;\/a&gt;\n\/\n&lt;a target=\"_blank\" href=\"\/people\/zhou-zhou-69-22\" class=\"zg-link-gray-normal\"&gt;1 \u8d5e\u540c&lt;\/a&gt;\n&lt;\/div&gt;\n\n&lt;\/div&gt;\n&lt;\/div&gt;","&lt;div class=\"zm-profile-card zm-profile-section-item zg-clear no-hovercard\"&gt;\n&lt;div class=\"zg-right\"&gt;\n&lt;button data-follow=\"m:button\" data-id=\"3a1a9da0e0bb4abe2554fa2a6032f27f\" class=\"zg-btn zg-btn-follow zm-rich-follow-btn small nth-0\"&gt;\u5173\u6ce8\u5979&lt;\/button&gt;\n&lt;\/div&gt;\n&lt;a title=\"\u7f8e\u7f8e\u836f\u5242\u5e08\"\ndata-hovercard=\"p$t$sui-nuo-81\"\nclass=\"zm-item-link-avatar\"\nhref=\"\/people\/sui-nuo-81\"&gt;\n&lt;img src=\"https:\/\/pic2.zhimg.com\/ae23b8e89725a24de650dee53e9a60a5_m.jpg\" class=\"zm-item-img-avatar\"&gt;\n&lt;\/a&gt;\n&lt;div class=\"zm-list-content-medium\"&gt;\n&lt;h2 class=\"zm-list-content-title\"&gt;&lt;span class=\"author-link-line\"&gt;\n&lt;a data-hovercard=\"p$t$sui-nuo-81\" href=\"https:\/\/www.zhihu.com\/people\/sui-nuo-81\" class=\"zg-link 
</code></pre>

<p>Unfortunately it is also invalid <em>json</em> so we cannot call <code>req.json()</code> and get nice un-escaped html so you will have to do it manually using <em>string_escape</em>:</p>

<pre><code>In [14]: rep = requests.post(url=zhihu_rl, data=data, headers=headers)

In [15]: bsobj = BeautifulSoup(rep.text.decode("string_escape"),  'lxml')

In [16]: ancs = (bsobj.find_all('div', {'class': 'zm-profile-card zm-profile-section-item zg-clear no-hovercard'}))

In [17]: len(ancs)
Out[17]: 20
</code></pre>

<p>It is also  <em><code>zm-profile-section-item</code></em> not  <em><code>zm-profile-section-   item</code></em></p>

<p>Also in future never post login cookies, I could have full access to your account in a couple of minutes.</p>

python beautifulsoup web-crawler