requests - fetch data from api-based website
<p>I want to get all the review from <a href="https://www.traveloka.com/hotel/singapore/mandarin-orchard-singapore-10602" rel="nofollow">this site</a>.</p>

<p>at first, I use this code:</p>

<pre><code>import requests
from bs4 import BeautifulSoup

r = requests.get(
    "https://www.traveloka.com/hotel/singapore/mandarin-orchard-singapore-10602")

data = r.content
soup = BeautifulSoup(data, "html.parser")
reviews = soup.find_all("div", {"class": "reviewText"})

for i in range(len(reviews)):
    print(reviews[i].get_text())
</code></pre>

<p>But this way, I can only get the reviews from the first page only.</p>

<p>Some said I could use api for this using the same <code>requests</code> module. I've found the api which is <a href="https://api.traveloka.com/v1/hotel/hotelReviewAggregate" rel="nofollow">https://api.traveloka.com/v1/hotel/hotelReviewAggregate</a> but I can't read the parameter because I don't know how to use api which use <code>request payload</code> way.</p>

<p>So I'm hoping for a code to get all the review using python or the parameter of api to get the review of specific hotel in all or specific pages.</p>

<p>Look at the request payload at the network tab. There is a part where <code>skip:8</code> and <code>top:8</code> and you will see those numbers increment by 8 when you click on the right arrow to get the next page of reviews. </p>

<p>You can duplicate that request and scrape the results the same way</p>

<p><strong>Edit:</strong></p>

<p>Open your page with chrome and hit <code>f12</code>. Go to <code>Network</code> tab, scroll down at the bottom of your page where you can advance to the next batch of reviews. As soon as you hit the right arrow the network tab will be populated. Find the second <code>hotelReviewAggregate</code> and click on it. Under the headers tab you will find <code>Request Payload</code>. Open the <code>data</code> dict and find <code>skip</code> and <code>top</code>. Advance the next batch of reviews and see how those numbers change. You can simulate this behavior to get to the other pages. </p>

<p>Then what you need to do is to prepare your payload where you can increment the values and make <code>GET</code> requests and use the <code>response objects</code> to scrape the data with BeautifulSoup.  </p>

<p>Requests <a href="http://docs.python-requests.org/en/master/user/quickstart/" rel="nofollow">here</a></p>

<p>Quick Example from the tutorial:</p>

<p><code>payload = {'key1': 'value1', 'key2': 'value2'}
r = requests.get('http://httpbin.org/get', params=payload)</code></p>

<p>I don't know why people decided to give a negative value to my answer without an explanation. But ohh well, If you find this useful and answers your question, please accept it.</p>

python python-2.7 api python-requests