Pulling Cuisine Type from TripAdvisor Restaurants using Python
<p>I am currently trying to pull data from TripAdvisor Restaurant from different countries. The fields I am trying to pull is name, address, and cuisine type (chinese, steakhouse, etc.). I have successfully been able to pull name and address using my script; however, pulling cuisine type is proving pretty difficult for myself. If you take a look below, you'll find screenshots of what I am trying to pull from TripAdvisor, and my code. </p>

<p><a href="https://i.stack.imgur.com/ajaFg.png" rel="nofollow">What I want to pull from TripAdvisor is circled in red.</a></p>

<p><a href="https://i.stack.imgur.com/qsGks.png" rel="nofollow">When I print my code it keeps printing 'Asian' even thought the second one should be a 'Steakhouse'.</a></p>

<pre><code>#import libraries
import requests
from bs4 import BeautifulSoup
import csv

#loop to move into the next pages. entries are in increments of 30 per page
for i in range(0, 120, 30):
    #need this here for when you want more than 30 entries pulled
    while i &lt;= range:
        i = str(i)
        #url format offsets the restaurants in increments of 30 after the oa
        url1 = 'https://www.tripadvisor.com/Restaurants-g294217-oa' + i + '-Hong_Kong.html#EATERY_LIST_CONTENTS'
        r1 = requests.get(url1)
        data1 = r1.text
        soup1 = BeautifulSoup(data1, "html.parser")
        for link in soup1.findAll('a', {'property_title'}):
            #print 'https://www.tripadvisor.com/Restaurant_Review-g294217-' + link.get('href')
            restaurant_url = 'https://www.tripadvisor.com/Restaurant_Review-g294217-' + link.get('href')
            #print link.string
            account_name = link.string.strip()
            #cuisine type pull
            for link in soup1.findAll('a', {'cuisine'}):
                cuisinetype = link.string.strip()
            r_address = requests.get(restaurant_url)
            r_addresstext = r_address.text
            soup2 = BeautifulSoup(r_addresstext, "html.parser")
            for restaurant_url in soup2.findAll('span', {'street-address'})[0]:
                #print(restaurant_url.string)
                rest_address = restaurant_url.string
                rest_array = [account_name, rest_address, cuisinetype]
                print rest_array
                #with open('ListingsPull-HongKong.csv', 'a') as file:
                    #writer = csv.writer(file)
                    #writer.writerow([account_name, rest_address])
        break
</code></pre>

<p>This approach is not especially elegant but might be acceptable to you. I notice that the information you want seems to be repeated under the 'Details' tab for 'Cuisine'. I've found it easier to access there this way.</p>

<pre><code>&gt;&gt;&gt; import requests
&gt;&gt;&gt; from bs4 import BeautifulSoup
&gt;&gt;&gt; restaurant_url='https://www.tripadvisor.ca/Restaurant_Review-g294217-d2399904-Reviews-Tin_Lung_Heen-Hong_Kong.html'
&gt;&gt;&gt; soup2 = BeautifulSoup(requests.get(restaurant_url).text, "html.parser")
&gt;&gt;&gt; street_address=soup2.find('span',{'street-address'})
&gt;&gt;&gt; street_address
&lt;span class="street-address" property="streetAddress"&gt;International Commerce Centre, 1 Austin Road West, Kowloon&lt;/span&gt;
&gt;&gt;&gt; street_address.contents[0]
'International Commerce Centre, 1 Austin Road West, Kowloon'
&gt;&gt;&gt; for item in soup2.findAll('div', attrs={'class', 'title'}):
...     if 'Cuisine' in item.text:
... 
...         item.text.strip()
...         break
...         
'Cuisine'
&gt;&gt;&gt; content=item.findNext('div', attrs={'class', 'content'})
&gt;&gt;&gt; content
&lt;div class="content"&gt;
Chinese,Â Asian
&lt;/div&gt;
&gt;&gt;&gt; content.text
'\nChinese,\xa0Asian\n'
&gt;&gt;&gt; content.text.strip().split('\xa0')
['Chinese,', 'Asian']
</code></pre>

python python-2.7 web-scraping beautifulsoup