collect text from web pages of a given site
<p>There is a site that I frequenly visit and read the "best advice". Here is how I can easily extract the text that I want...</p>

<pre><code>import urllib2
from bs4 import BeautifulSoup  

mylist=list()

myurl='http://www.apartmenttherapy.com/carols-east-side-cottage-house-tour-194787'
s=urllib2.urlopen(myurl)
soup =  BeautifulSoup(s)

hello = soup.find(text='Best Advice: ')
mylist.append(hello.next)
</code></pre>

<p>But how do I collect the text snippets from all the pages?</p>

<hr>

<p>I can search for all pages using this simple google query...</p>

<p>site:<a href="http://www.apartmenttherapy.com" rel="nofollow">http://www.apartmenttherapy.com</a></p>

<p>Does google search has API that can be used in python?
I am looking for one time simple solution for this problem. So I will prefer not to install too many packages to get this task done.</p>

<p>You have to use a js-enabled scraping like the way explained here : 
<a href="http://koaning.io/dynamic-scraping-with-python.html" rel="nofollow">http://koaning.io/dynamic-scraping-with-python.html</a></p>

python beautifulsoup google-search-api