Scrapy - processing items with pipeline
<p>I'm running <code>scrapy</code>from a <code>python script</code>.</p>

<p>I was told that in <code>scrapy</code>, <code>responses</code> are built in <code>parse()</code>and further processed in <code>pipeline.py</code>. </p>

<p>this is how my <code>framework</code> is set so far:</p>

<p><strong>python script</strong></p>

<pre><code>def script(self):

        process = CrawlerProcess(get_project_settings())

        response = process.crawl('pitchfork_albums', domain='pitchfork.com')

        process.start() # the script will block here until the crawling is finished
</code></pre>

<p><strong>spiders</strong></p>

<pre><code>class PitchforkAlbums(scrapy.Spider):
    name = "pitchfork_albums"
    allowed_domains = ["pitchfork.com"]
    #creates objects for each URL listed here
    start_urls = [
                    "http://pitchfork.com/reviews/best/albums/?page=1",
                    "http://pitchfork.com/reviews/best/albums/?page=2",
                    "http://pitchfork.com/reviews/best/albums/?page=3"                   
    ]
    def parse(self, response):

        for sel in response.xpath('//div[@class="album-artist"]'):
            item = PitchforkItem()
            item['artist'] = sel.xpath('//ul[@class="artist-list"]/li/text()').extract()
            item['album'] = sel.xpath('//h2[@class="title"]/text()').extract()

        yield item
</code></pre>

<p><strong>items.py</strong></p>

<pre><code>class PitchforkItem(scrapy.Item):

    artist = scrapy.Field()
    album = scrapy.Field()
</code></pre>

<p><strong>settings.py</strong></p>

<pre><code>ITEM_PIPELINES = {
   'blogs.pipelines.PitchforkPipeline': 300,
}
</code></pre>

<p><strong>pipelines.py</strong> </p>

<pre><code>class PitchforkPipeline(object):

    def __init__(self):
        self.file = open('tracks.jl', 'wb')

    def process_item(self, item, spider):
        line = json.dumps(dict(item)) + "\n"
        self.file.write(line)
        for i in item:
            return i['album'][0]
</code></pre>

<p>if I just <code>return item</code> in <code>pipelines.py</code>, I get data like so (one <code>response</code> for each <code>html</code>page):</p>

<pre><code>{'album': [u'Sirens',
           u'I Had a Dream That You Were Mine',
           u'Sunergy',
           u'Skeleton Tree',
           u'My Woman',
           u'JEFFERY',
           u'Blonde / Endless',
           u' A Mulher do Fim do Mundo (The Woman at the End of the World) ',
           u'HEAVN',
           u'Blank Face LP',
           u'blackSUMMERS\u2019night',
           u'Wildflower',
           u'Freetown Sound',
           u'Trans Day of Revenge',
           u'Puberty 2',
           u'Light Upon the Lake',
           u'iiiDrops',
           u'Teens of Denial',
           u'Coloring Book',
           u'A Moon Shaped Pool',
           u'The Colour in Anything',
           u'Paradise',
           u'HOPELESSNESS',
           u'Lemonade'],
 'artist': [u'Nicolas Jaar',
            u'Hamilton Leithauser',
            u'Rostam',
            u'Kaitlyn Aurelia Smith',
            u'Suzanne Ciani',
            u'Nick Cave &amp; the Bad Seeds',
            u'Angel Olsen',
            u'Young Thug',
            u'Frank Ocean',
            u'Elza Soares',
            u'Jamila Woods',
            u'Schoolboy Q',
            u'Maxwell',
            u'The Avalanches',
            u'Blood Orange',
            u'G.L.O.S.S.',
            u'Mitski',
            u'Whitney',
            u'Joey Purp',
            u'Car Seat Headrest',
            u'Chance the Rapper',
            u'Radiohead',
            u'James Blake',
            u'White Lung',
            u'ANOHNI',
            u'Beyonc\xe9']}
</code></pre>

<p>what I would like to do in <code>pipelines.py</code>is to be able to fetch individual <code>songs</code> for each <code>item</code>, like so:</p>

<pre><code>[u'Sirens']
</code></pre>

<p>please help?</p>

<p>I suggest that you build well structured <code>item</code> in spider. In Scrapy Framework work flow, spider is used to built well-formed item, e.g., parse html, populate item instances and pipeline is used to do operations on item, e.g., filter item, store item.</p>

<p>For your application, if I understand correctly, each item should be an entry to describe an album. So when paring html, you'd better build such kind of item, instead of massing everything into item.</p>

<p>So in your <code>spider.py</code>, <code>parse</code> function, you should </p>

<ol>
<li><strong>Put <code>yield item</code> statement in the <code>for</code> loop, NOT OUTSIDE</strong>. In this way, each album will generate an item.</li>
<li><strong>Be careful about relative xpath selector in Scrapy</strong>. If you want to use relative xpath selector to specify self-and-descendant, use <code>.//</code> instead of <code>//</code>, and to specify self, use <code>./</code> instead of <code>/</code>.</li>
<li><p>Ideally album title should be a scalar, album artist should be a list, so try <code>extract_first</code> to make album title to be a scalar.</p>

<pre><code>def parse(self, response):
for sel in response.xpath('//div[@class="album-artist"]'):
    item = PitchforkItem()
    item['artist'] = sel.xpath('./ul[@class="artist-list"]/li/text()').extract_first()
    item['album'] = sel.xpath('./h2[@class="title"]/text()').extract()
    yield item
</code></pre></li>
</ol>

<p>Hope this would be helpful.</p>

python scrapy