Accessing dynamic generated website content with Python requests
<p>I'm trying to gather data from few websites using Python (BeautifulSoup). However, sometimes it's difficult to access search results, example:</p>

<pre><code>import requests
from bs4 import BeautifulSoup

url1 = 'https://auto.ria.com/legkovie/city/vinnica/?page=1'
url2= 'https://auto.ria.com/search/?top=11&amp;category_id=1&amp;state[0]=1'

def get_value(url):
    r = requests.get(url, headers = {'Accept-Encoding' : 'deflate'})
    print("Response Time: {}".format(r.elapsed.total_seconds()))

    soup =  BeautifulSoup(r.text, 'lxml')
    data = soup.find('span', attrs = {'id' : 'resultsCount'}).find('strong')
    print('{} \n'.format(data))

get_value(url1)
get_value(url2)
</code></pre>

<p>The output is:</p>

<pre><code>Response Time: 5.4943
&lt;strong class="count"&gt;5 310&lt;/strong&gt; 

Response Time: 0.174867
&lt;strong class="count"&gt;0&lt;/strong&gt;  
</code></pre>

<p>though in case of url2 the number displayed in browser is 338.
I suppose that search results can be found in some json, but how to access it using requests?</p>

<p>I recommend zooming out on the detail of the soup object to see what's there. You can try using findAll instead of find and printing results. You can also try stripping the final call to find (for the strong tag) and print the results. Once you investigate the larger object you'll likely see what is happening. It could be url2 is tagged differently and you'll have to adjust your function to accommodate. </p>

python json web-scraping python-requests