Accessing dynamic generated website content with Python requests
<p>I'm trying to gather data from few websites using Python (BeautifulSoup). However, sometimes it's difficult to access search results, example:</p>

<pre><code>import requests
from bs4 import BeautifulSoup

url1 = 'https://auto.ria.com/legkovie/city/vinnica/?page=1'
url2= 'https://auto.ria.com/search/?top=11&amp;category_id=1&amp;state[0]=1'

def get_value(url):
    r = requests.get(url, headers = {'Accept-Encoding' : 'deflate'})
    print("Response Time: {}".format(r.elapsed.total_seconds()))

    soup =  BeautifulSoup(r.text, 'lxml')
    data = soup.find('span', attrs = {'id' : 'resultsCount'}).find('strong')
    print('{} \n'.format(data))

get_value(url1)
get_value(url2)
</code></pre>

<p>The output is:</p>

<pre><code>Response Time: 5.4943
&lt;strong class="count"&gt;5 310&lt;/strong&gt; 

Response Time: 0.174867
&lt;strong class="count"&gt;0&lt;/strong&gt;  
</code></pre>

<p>though in case of url2 the number displayed in browser is 338.
I suppose that search results can be found in some json, but how to access it using requests?</p>

<p>Your code is running fine, and url2 returns the expected result. By viewing page source from chrome:<br>
<em><code>&lt;span id="resultsCount" class="hide"&gt;ÐÐ°Ð¹Ð´ÐµÐ½Ð¾ &lt;strong class="count"&gt;0&lt;/strong&gt; Ð¾Ð±ÑÑÐ²Ð»ÐµÐ½Ð¸Ð¹&lt;/span&gt;</code></em></p>

<p>This is the tag that your are trying to find with beautiful soup. The number displayed in chrome and the program's output are the same!</p>

<pre><code>&lt;strong class="count"&gt;0&lt;/strong&gt; 
</code></pre>

<p>Also, the search result is not returned in json. If you check the response headers:  </p>

<pre><code>Content-Type: text/html
</code></pre>

<p>Maybe you want the response to contain the whole tag instead? If that's the case, try:  </p>

<pre><code>data = soup.find('span', attrs = {'id' : 'resultsCount'})  
</code></pre>

python json web-scraping python-requests