IndexError when using Scrapy for absolute links
<p>I am scraping a webpage from Wikipedia (particularly <a href="https://en.wikipedia.org/wiki/Category:2013_films" rel="nofollow">this</a> one) using a Python library called <a href="https://doc.scrapy.org/en/1.2/" rel="nofollow">Scrapy</a>. Here was the original code which successfully crawled the page:</p>

<pre><code>import scrapy
from wikipedia.items import WikipediaItem


class MySpider(scrapy.Spider):
    name = "wiki"
    allowed_domains = ["en.wikipedia.org/"]
    start_urls = [
        'https://en.wikipedia.org/wiki/Category:2013_films',
    ]

    def parse(self, response):
        titles = response.xpath('//div[@id="mw-pages"]//li')
        items = []
        for title in titles:
            item = WikipediaItem()
            item["title"] = title.xpath("a/text()").extract()
            item["url"] = title.xpath("a/@href").extract()
            items.append(item)
        return items
</code></pre>

<p>Then in the terminal, I ran <code>scrapy crawl wiki -o wiki.json -t json</code> to output the data to a JSON file. While the code worked, the links assigned to the "url" keys were all relative links. (i.e.: <code>{"url": ["/wiki/9_Full_Moons"], "title": ["9 Full Moons"]}</code>).</p>

<p>Instead of <em>/wiki/9_Full_Moons</em>, I needed <em><a href="http://en.wikipedia.org/wiki/9_Full_Moons" rel="nofollow">http://en.wikipedia.org/wiki/9_Full_Moons</a></em>. So I modified the above mentioned code to import the <strong>urljoin</strong> from the urlparse library. I also modified my <code>for</code> loop to look like this instead:</p>

<pre><code>for title in titles:
    item = WikipediaItem()
    url = title.xpath("a/@href").extract()
    item["title"] = title.xpath("a/text()").extract()
    item["url"] = urljoin("http://en.wikipedia.org", url[0])
    items.append(item)
return(items)
</code></pre>

<p>I believed this was the correct approach since the type of data assigned to the <code>url</code> key is enclosed in brackets (which would entail a list, right?) so to get the string inside it, I typed <strong>url[0]</strong>. However, this time I got an IndexError that looked like this:</p>

<blockquote>
  <p>IndexError: list index out of range</p>
</blockquote>

<p>Can someone help explain where I went wrong?</p>

<p>I think you can just concatenate the two strings instead of using <code>urljoin</code>. Try this:</p>

<pre><code>for title in titles:
    item = WikipediaItem()
    item["title"] = title.xpath("a/text()").extract()
    item["url"] = "http://en.wikipedia.org" + title.xpath("a/@href").extract()[0]
    items.append(item)
return(items)
</code></pre>

python json xpath scrapy