Request returns partial page
<p>I'm trying to parse data of a website that loads when user scroll. There is a finite number of element that can appears while scrolling, but using this only gives the first part (25 out of 112):</p>

<pre><code>url = "http://url/to/website"
response = requests.get(url)
soup = BeautifulSoup(response.text)
</code></pre>

<p>How can I tell <code>request</code> to "scroll" before returning the html?</p>

<p>EDIT : apparently request don't do that, what solution can I use in Python? </p>

<blockquote>
  <p>The only thing you should know is how serverlet works.</p>
</blockquote>

<p>Usually, <code>onScroll</code> or <code>onClick</code> or any other event will trigger <code>AJAX request</code> to the server. And the client side javascript will render those return (JSON/XML...) So the only thing you should do is to repeat those AJAX request to the same server to get those data.</p>

<p>For example, the action in browser will like below:</p>

<pre><code>1. Enter url on browser   
&gt; [HTTP GET REQUEST] http://url/to/website

2. Scroll on the page
&gt; [AJAX GET] http://url/to/website/1
&gt; [javascript on front-end will process those data]

3. Then, keeping scrolling on the page
&gt; [AJAX GET] http://url/to/website/2
&gt; [javascript on front-end will process those data]

4. ... (and so on)
</code></pre>

<hr>

<p><strong>Q. How to use python to get those data?</strong></p>

<p>A. One simple way is using <code>browser &gt; inspect &gt; network_tab</code> to find what AJAX request you send when you scroll in that page. And repeat those AJAX request with correspond header by python.</p>

python beautifulsoup python-requests