Webscraping with Python ( beginner)
<p>I'm doing the first example of the webscrapping tutorial from the book "Automate the Boring Tasks with Python". The project consists of typing a search term on the command line and have my computer automatically open a browser with all the top search results in new tabs</p>

<p>It mentions that I need to locate the </p>

<pre><code>&lt;h3 class="r"&gt; 
</code></pre>

<p>element from the page source, which are the links to each search results. 
The r class is used only for search result links.</p>

<p>But the problem is that I can't find it anywhere, even using Chrome Devtools. Any help as to where is it would be greatly appreciated.</p>

<p>Note: Just for reference this is the complete program as seen on the book.</p>

<pre><code># lucky.py - Opens several Google search results.

import requests, sys, webbrowser, bs4

print('Googling..') # display text while downloading the Google page
res= requests.get('http://google.com/search?q=' + ' '.join(sys.argv[1:]))
res.raise_for_status()

#Retrieve top searh result links.
soup = bs4.BeautifulSoup(res.text)

#Open a browser tab for each result.
linkElems = soup.select('.r a')
numOpen = min(5,len(linkElems))
for i in range(numOpen):
    webbrowser.open('http://google.com' + linkElems[i].get('href'))
</code></pre>

<p>This will work for you : </p>

<pre><code>&gt;&gt;&gt; import requests
&gt;&gt;&gt; from lxml import html
&gt;&gt;&gt; r = requests.get("https://www.google.co.uk/search?q=how+to+do+web+scraping&amp;num=10")
&gt;&gt;&gt; source = html.fromstring((r.text).encode('utf-8'))
&gt;&gt;&gt; links = source.xpath('//h3[@class="r"]//a//@href')
&gt;&gt;&gt; for link in links:
        print link.replace("/url?q=","").split("&amp;sa=")[0]
</code></pre>

<p>Output :</p>

<pre><code>http://newcoder.io/scrape/intro/
https://www.analyticsvidhya.com/blog/2015/10/beginner-guide-web-scraping-beautiful-soup-python/
http://docs.python-guide.org/en/latest/scenarios/scrape/
http://webscraper.io/
https://blog.hartleybrody.com/web-scraping/
https://first-web-scraper.readthedocs.io/
https://www.youtube.com/watch%3Fv%3DE7wB__M9fdw
http://www.gregreda.com/2013/03/03/web-scraping-101-with-python/
http://analystcave.com/web-scraping-tutorial/
https://en.wikipedia.org/wiki/Web_scraping
</code></pre>

<p><strong>Note</strong>: <em>I am using Python 2.7.X , for Python 3.X you just have to surround the print output like this</em> <strong>print (link.replace("/url?q=","").split("&amp;sa=")[0])</strong></p>

python web-scraping