working with "load more" button using scrapy
<p>I was working on ratemyprofessor which has a load more button to load more professor, and I use debugger to analyze network, it shows up a js request. 
<a href="https://www.ratemyprofessors.com/campusRatings.jsp?sid=1273" rel="nofollow">ratemyprofessorwebsite</a></p>

<p>I was thinking, for the request URL, there is a start and rows, so I just increase the start by 20 each time, will this work? </p>

<p>And someone told me that I could try formdata, but in this case, there is no formdata and it is not a POST method, am I right?</p>

<p>I am really new to scrapy and python, hope u guys can give me some insight. Really appreciate it </p>

<p>they don't allow me to upload image... but anyway</p>

<pre><code>Request URL:https://search-a.akamaihd.net/typeahead/suggest/?solrformat=true&amp;rows=10&amp;callback=noCB&amp;q=*%3A*+AND+schoolid_s%3A1273&amp;defType=edismax&amp;qf=teacherfullname_t%5E1000+autosuggest&amp;bf=pow(total_number_of_ratings_i%2C2.1)&amp;sort=total_number_of_ratings_i+desc&amp;siteName=rmp&amp;rows=20&amp;start=20&amp;fl=pk_id+teacherfirstname_t+teacherlastname_t+total_number_of_ratings_i+averageratingscore_rf+schoolid_s
Request Method:GET
Status Code:200 OK
Remote Address:23.212.53.206:443
</code></pre>

<p>QUERY STING PARAMETERS</p>

<pre><code>solrformat:true
rows:10
callback:noCB
q:*:* AND schoolid_s:1273
defType:edismax
qf:teacherfullname_t^1000 autosuggest
bf:pow(total_number_of_ratings_i,2.1)
sort:total_number_of_ratings_i desc
siteName:rmp
rows:20
start:20
fl:pk_id teacherfirstname_t teacherlastname_t total_number_of_ratings_iaverageratingscore_rf schoolid_s
</code></pre>

<p>Yes, you are correct - formdata is not required, and this is straight forward using GET method calls.</p>

<p>using the following parameters, grab the json response with:</p>

<pre><code>data = json.loads(response.body)
records = data['response']['docs'] 
</code></pre>

<hr>

<p>parameters to use : <strong>start, rows</strong></p>

<p>parameter to avoid : <strong>callback</strong></p>

<p>parameters with explanation:</p>

<pre><code>solrformat:true
#callback:noCB # Drop this parameter, this will give you response data in json format
q:*:* AND schoolid_s:1273
defType:edismax
qf:teacherfullname_t^1000 autosuggest
bf:pow(total_number_of_ratings_i,2.1)
sort:total_number_of_ratings_i desc
siteName:rmp
rows:20 
# rows: the amount of records you want in a single response, this will give you 20
# you can try with maximum that can be returned in single response i.e 500, 1000..etc, this will minimize no of request made to website 
start:20  
#start :you can start this from 0,10,20,30,40 manually as well if you know the total number of records.
fl:pk_id teacherfirstname_t teacherlastname_t total_number_of_ratings_iaverageratingscore_rf schoolid_s
</code></pre>

python scrapy