scrapy response.xpath only picking out the first item
<p>I have the html structure</p>

<pre><code>  &lt;div class="column first"&gt;
    &lt;div class="detail"&gt;
      &lt;strong&gt;Phone: &lt;/strong&gt;
      &lt;span class="value"&gt; 012-345-6789&lt;/span&gt;
    &lt;/div&gt;
    &lt;div class="detail"&gt;
      &lt;span class="value"&gt;1 Street Address, Big Road, City, Country&lt;/span&gt;
    &lt;/div&gt;
    &lt;div class="detail"&gt;
      &lt;h3 class="inline"&gt;Area:&lt;/h3&gt;
      &lt;span class="value"&gt;Georgetown&lt;/span&gt;
    &lt;/div&gt;
    &lt;div class="detail"&gt;
      &lt;h3 class="inline"&gt;Nearest Train:&lt;/h3&gt;
      &lt;span class="value"&gt;Georgetown Station&lt;/span&gt;
    &lt;/div&gt;
    &lt;div class="detail"&gt;
      &lt;h3 class="inline"&gt;Website:&lt;/h3&gt;
      &lt;span class="value"&gt;&lt;a href='http://www.website.com' target='_blank'&gt;www.website.com&lt;/a&gt;&lt;/span&gt;
    &lt;/div&gt;
  &lt;/div&gt;
</code></pre>

<p>When I run <code>sel = response.xpath('//span[@class="value"]/text()')</code> in scrapy shell I get what I expect back, which is:</p>

<pre><code>[&lt;Selector xpath='//span[@class="value"]/text()' data=u' 012-345-6789'&gt;, &lt;Selector xpath='//span[@class="value"]/text()' data=u'1 Street Address, Big Road, City, Country'&gt;, &lt;Selector xpath='//span[@class="value"]/text()' data=u'Georgetown Station'&gt;, &lt;Selector xpath='//span[@class="value"]/text()' data=u' '&gt;, &lt;Selector xpath='//span[@class="value"]/text()' data=u'January, 2016'&gt;]
</code></pre>

<p>However, in the parse block in my scrapy spider, it's only returning the first item</p>

<pre><code>def parse(self, response):
    def extract_with_xpath(query):
        return response.xpath(query).extract_first().strip()

    yield {
        'details': extract_with_xpath('//span[@class="value"]/text()')
    }
</code></pre>

<p>I realise I am using <code>extract_first()</code> but if I use <code>extract()</code> it breaks, even though I know <code>extract()</code> is a legitimate function.</p>

<p>What I am doing wrong? Do I need to loop through the 
<code>extract_with_xpath('//span[@class="value"]/text()')</code> part?</p>

<p>Thanks for any enlightenment!</p>

<p>in items.py, specify-</p>

<pre><code>from scrapy.item import Item, Field

class yourProjectNameItem(Item):
    # define the fields for your item here like:
    name = Field()
    details= Field()
</code></pre>

<p>in your scrapy spider:
imports:</p>

<pre><code>from scrapy.selector import HtmlXPathSelector
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.contrib.spiders import CrawlSpider, Rule
from yourProjectName.items import yourProjectNameItem
import re
</code></pre>

<p>and the parse function as follows:</p>

<pre><code>def parse_item(self, response):
    hxs = HtmlXPathSelector(response)
    i = yourProjectNameItem()

    i['name'] = hxs.select('YourXPathHere').extract() 
    i['details'] = hxs.select('YourXPathHere').extract()

    return i
</code></pre>

<p>Hope this solves the issue.
You can refer to my project on git:<a href="https://github.com/omkar-dsd/SRMSE/tree/master/Scrapers/NasaScraper" rel="nofollow">https://github.com/omkar-dsd/SRMSE/tree/master/Scrapers/NasaScraper</a></p>

python scrapy