Scrapy - iterate over object
<p>this is how I'm running <code>scrapy</code> from a <code>Python</code> script:</p>

<pre><code>def iterate():

    process = CrawlerProcess(get_project_settings())

    tracks = process.crawl('pitchfork_tracks', domain='pitchfork.com')

    process.start() 
</code></pre>

<p>however, I can't seem to <code>iterate</code> through the <code>response</code>, which is a a <code>dict</code> in this fashion:</p>

<pre><code>{'track': [u'\u201cAnxiety\u201d',
           u'\u201cLockjaw\u201d [ft. Kodak Black]',
           u'\u201cMelanin Drop\u201d',
           u'\u201cDreams\u201d',
           u'\u201cIntern\u201d',
           u'\u201cYou Don\u2019t Think You Like People Like Me\u201d',
           u'\u201cFirst Day Out tha Feds\u201d',
           u'\u201cFemale Vampire\u201d',
           u'\u201cGirlfriend\u201d',
           u'\u201cOpposite House\u201d',
           u'\u201cGirls @\u201d [ft. Chance the Rapper]',
           u'\u201cI Am a Nightmare\u201d']}
</code></pre>

<p>how do I <code>iterate</code> through this <code>response</code>? To my knowledge, up to this point the response it is an <code>object</code> and thus non-iterable.</p>

<p>You should follow the work flow of Scrapy Framework. Spider handles how requests are built and responses are parsed. ItemPipeline handles how items are operated.</p>

<p>From your code:</p>

<pre><code>tracks = process.crawl('pitchfork_tracks', domain='pitchfork.com')
</code></pre>

<p><code>pitchfork_tracks</code> is a spider name in your project. So you should handle the iteration on response in the spider, and do further operation in itempipeline. For the ItemPipeline part, you need to manually configure the settings of scrapy script. Check the docs for running scrapy from script approaches <a href="https://doc.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script" rel="nofollow">common practice-run from script</a></p>

<p>By the way, according to the docs <a href="https://doc.scrapy.org/en/latest/topics/api.html#scrapy.crawler.CrawlerProcess" rel="nofollow">CrawlerProcess</a>,</p>

<p>tracks = process.crawl('pitchfork_tracks', domain='pitchfork.com')</p>

<p><code>tracks</code> is a twist <code>defer</code> object, and this object is not iterable. Unless you are familiar with twist and Scrapy internal part, you'd better follow work flow of Scrapy framework.</p>

<p>Thanks.</p>

python scrapy iteration