Receiving and empty list when trying to make a webscraper to parse websites for links
<p>I was reading <a href="http://docs.python-guide.org/en/latest/scenarios/scrape/" rel="nofollow">this</a> website and learning how to make a webscraper with <code>lxml</code> and `Requests. This is the webscraper code:</p>

<pre><code>from lxml import html
import requests

web_page = requests.get('http://econpy.pythonanywhere.com/ex/001.html')

tree = html.fromstring(web_page.content)


buyers = tree.xpath('//div[@title="buyer-name"]/text()')
prices = tree.xpath('//span[@class="item-price"]/text()')

print "These are the buyers: ", buyers
print "And these are the prices: ", prices 
</code></pre>

<p>It works as intended, but when I try to scrape <a href="https://www.reddit.com/r/cringe/" rel="nofollow">https://www.reddit.com/r/cringe/</a> for all the links I'm simply getting <code>[]</code> as a result: </p>

<pre><code>#this code will scrape a Reddit page

from lxml import html
import requests


web_page = requests.get("https://www.reddit.com/r/cringe/")

tree = html.fromstring(web_page.content)

links = tree.xpath('//div[@class="data-url"]/text()')

print links
</code></pre>

<p>What's the problem with the xpath I'm using? I can't figure out what to put in the square brackets in the xpath</p>

<p>First off, your xpath is wrong, there are no classes with <em>data-url</em>, it is an <em>attribute</em> so you would want <code>div[@data-url]</code> and to extract the attribute you would use  <code>/@data-url</code>:</p>

<pre><code>from lxml import html
import requests

headers = {"User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.92 Safari/537.36"}
web_page = requests.get("https://www.reddit.com/r/cringe/", headers=headers)

tree = html.fromstring(web_page.content)

links = tree.xpath('//div[@data-url]/@data-url')

print links
</code></pre>

<p>Also you may see html like the following returned if you query too often or don't use a user-agent so respect what they recommend:</p>

<pre><code>&lt;p&gt;we're sorry, but you appear to be a bot and we've seen too many requests
from you lately. we enforce a hard speed limit on requests that appear to come
from bots to prevent abuse.&lt;/p&gt;

&lt;p&gt;if you are not a bot but are spoofing one via your browser's user agent
string: please change your user agent string to avoid seeing this message
again.&lt;/p&gt;

&lt;p&gt;please wait 6 second(s) and try again.&lt;/p&gt;

    &lt;p&gt;as a reminder to developers, we recommend that clients make no
    more than &lt;a href="http://github.com/reddit/reddit/wiki/API"&gt;one
    request every two seconds&lt;/a&gt; to avoid seeing this message.&lt;/p&gt;
  &lt;/body&gt;
&lt;/html&gt;
</code></pre>

<p>If you plan on scraping a lot of reddit, you may want to look at <a href="https://praw.readthedocs.io/en/stable/" rel="nofollow">PRAW</a> and <a href="http://www.w3schools.com/xsl/xpath_syntax.asp" rel="nofollow">w3schools</a> has a nice introduction to <em>xpath</em> expressions.</p>

<p>To break it down:</p>

<pre><code>//div[@data-url]
</code></pre>

<p>searches the doc for <em>div's</em> that have an attribute <code>data-url</code> we don't care what the attribute value is, we just want the div. </p>

<p>That just finds the <em>div's</em>, if you removed the <em>/@data-url</em> you would end up with a list of elements like:</p>

<pre><code>[&lt;Element div at 0x7fbb27a9a940&gt;, &lt;Element div at 0x7fbb27a9a8e8&gt;,..
</code></pre>

<p><code>/@data-url</code>  actually extracts the <em>attrbute value</em> i.e the <em>hrefs</em>.</p>

<p>Also  you just wanted specific links, the <em>youtube</em> links you could filter using <em>contains</em>: </p>

<pre><code>'//div[contains(@data-url, "www.youtube.com")]/@data-url'
</code></pre>

<p><code>contains(@data-url, "www.youtube.com")</code> will check if the <em>data-url</em> attribute values contain <em>www.youtube.com</em> so the output will be a list of the <em>youtube</em> links.</p>

python html web-scraping lxml