Request returns partial page
<p>I'm trying to parse data of a website that loads when user scroll. There is a finite number of element that can appears while scrolling, but using this only gives the first part (25 out of 112):</p>

<pre><code>url = "http://url/to/website"
response = requests.get(url)
soup = BeautifulSoup(response.text)
</code></pre>

<p>How can I tell <code>request</code> to "scroll" before returning the html?</p>

<p>EDIT : apparently request don't do that, what solution can I use in Python? </p>

<p>You can't. The question is based on a misunderstanding of what requests does; it loads the content of the page only. Endless scrolling is powered by Javascript, which requests won't do anything with.</p>

<p>You'd need some browser automation tools like Selenium to do this; or find out what Ajax endpoint the scrolling JS is using and load that directly.</p>

python beautifulsoup python-requests