How to create a web crawler to get multiple pages from agoda,with python3
<p>I'm new to here. Recently, I want to get data from Agoda,and I got a problem that agoda.com don't provide the url(or href) of "next page". So I have no idea to change page. Now, I only get the data from page 1, but I need the data from page2, page3...
Is anyone help me. I need some advise, tools or others. By the way, I use python3 and win10.Please help me and thank you.
Below is my code presently.</p>

<pre><code>import requests
import pandas as pd

import csv
from bs4 import BeautifulSoup
from pandas import Series,DataFrame
import unicodecsv

def name1():
    url="https://www.agoda.com/zh-tw/pages/agoda/default/DestinationSearchResult.aspx?asq=%2bZePx52sg5H8gZw3pGCybdmU7lFjoXS%2baxz%2bUoF4%2bbAw3oLIKgWQqUpZ91GacaGdIGlJ%2bfxiotUg7cHef4W8WIrREFyK%2bHWl%2ftRKlV7J5kUcPb7NK6DnLacMaVs1qlGagsx8liTdosF5by%2fmvF3ZvJvZqOWnEqFCm0staf3OvDRiEYy%2bVBJyLXucnzzqZp%2fcBP3%2bKCFNOTA%2br9ARInL665pxj%2fA%2bylTfAGs1qJCjm9nxgYafyEWBFMPjt2sg351B&amp;city=18343&amp;cid=1732641&amp;tag=41460a09-3e65-d173-1233-629e2428d88e&amp;gclid=Cj0KEQjwvve_BRDmg9Kt9ufO15EBEiQAKoc6qlyYthgdt9CgZ7a6g6yijP42n6DsCUSZXvtfEJdYqiAaAvdW8P8HAQ&amp;tick=636119092231&amp;isdym=true&amp;searchterm=%E5%A2%BE%E4%B8%81&amp;pagetypeid=1&amp;origin=TW&amp;cid=1732641&amp;htmlLanguage=zh-tw&amp;checkIn=2016-10-20&amp;checkOut=2016-10-21&amp;los=1&amp;rooms=1&amp;adults=2&amp;children=0&amp;isFromSearchBox=true&amp;ckuid=1b070b17-86c2-4376-a4f5-d3b98fc9cf45" 
    source_code=requests.get(url)                                         
    plain_text=source_code.text                                           
    soup=BeautifulSoup(plain_text,"lxml")                                        
    hotelname=soup.find_all("h3",{"class":"hotel-name"})
    f = csv.writer(open("test.csv", "w",newline=''))
    f.writerow(["hotelname","address"]) 
    p = []
    for N in hotelname:
        a=N.string.strip()
        f.writerow([a])
</code></pre>

<p>Examine closely in browser development tools what happens when you click next button. </p>

<p>It has click event that sends xhr post request with a lot of parameters. One of the parameters is <code>PageNumber</code>. Most values for the parameters are straightforward to get, maybe except <code>SearchMessageID</code> that you have to find somewhere on page or is generated by javascript. </p>

python scrapy web-crawler webpage