Why is this python web scraping code using requests package not working?
<pre><code>import lxml.html  
import requests  
l1=[]  
headers= {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36'}  
r = requests.get('http://www.naukri.com/jobs-by-location', headers=headers)    
html = r.content  
root = lxml.html.fromstring(html)  
urls = root.xpath('//div[4]/div/div[1]/div/a/@href') #This xpath should give the list of cities(their links)  
l1.extend(urls)     
</code></pre>

<p>This python code is meant to scrape the list of job cities(their 'a href' tags) and store it in list l1. But here I am getting a blank list. The same xpath is working on Chrome console but it's not working in this code. Due to that I added headers to make my code act as a browser but still it's not working..  </p>

<p><a href="http://i.stack.imgur.com/Xx1xW.jpg" rel="nofollow">http://i.stack.imgur.com/Xx1xW.jpg</a></p>

<p>I tried to achieve the same using Selenium WebDriver, and this also succeeds. When this succeeds from your computer, it might be a problem in one of the used libraries.</p>

<pre><code>import selenium.webdriver as driver

browser = driver.Chrome()
browser.get("http://www.naukri.com/jobs-by-location")
links = browser.find_elements_by_xpath("//div[4]/div/div[1]/div/a")
for link in links:
    href = link.get_attribute("href")
    print(href)
browser.quit()
</code></pre>

python python-2.7 web-scraping python-requests