Python - Scrapy data lists
<p>I have the following piece of code in my scraper:</p>

<pre><code>import scrapy
import os
import re
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class MySpider(CrawlSpider):
    handle_httpstatus_list = [301,302,404,200,500]
    name = 'rust'
    allowed_domains = ['example.com']
    start_urls = ['http://example.com']

rules = (
    # Extract links matching 'category.php' (but not matching 'subsection.php')
    # and follow links from them (since no callback means follow=True by default).
    # Extract links matching 'item.php' and parse them with the spider's method parse_item
    Rule(LinkExtractor(), callback='parse_item', follow=True),
)

def parse_item(self, response):
 a = []
 if response.url == "http://example.com/":
  rlink = response.xpath('//a/@href').extract()
  litarget = response.xpath('//a/@target').extract()
  lirel = response.xpath('//a/@rel').extract()
  litext = response.xpath('//a/text()').extract()
  #print rlink
  for i, z in zip(rlink, litarget):
   #print i
   if i == "/some/link/":
    print z
</code></pre>

<p>This doesn't work for me :( </p>

<p>I want to achieve the following thing:
Since the extracted data is in lists: <code>rlink</code>, <code>litarget</code>, <code>lirel</code> and <code>litext</code>. I want to be able to extract corresponding information based on another one from the lists: </p>

<pre><code>if link in response.xpath('//a/@href').extract() == "/some/link"
</code></pre>

<blockquote>
  <p>print its target, rel and text attribute.</p>
</blockquote>

<p>Can you help me solve that.</p>

<p>Thanks!</p>

<p>You probably run your scrapy spider from command line.</p>

<p>In that case I would suggest you to debug your spider using pycharm ide.</p>

<p>Just add this code inside <code>yourproject</code> directory and name it something like  <code>main.py</code></p>

<pre><code># -*- coding: utf-8 -*-
import logging

from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
from scrapy.utils.project import get_project_settings
from twisted.internet import reactor, defer

from yourproject.spiders.my_spider import MySpider


configure_logging(install_root_handler=False)
logging.basicConfig(
    filename='log.txt',
    filemode='w',
    format='%(asctime)s: %(levelname)s: %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S',
    level=logging.DEBUG
)

console = logging.StreamHandler()
console.setLevel(logging.DEBUG)

# uncomment this line to print logs in console
#logging.getLogger('').addHandler(console)

logger = logging.getLogger(__name__)

settings=get_project_settings()

runner = CrawlerRunner(settings=settings)

@defer.inlineCallbacks
def crawl():
    yield runner.crawl(MySpider)
    reactor.stop()

crawl()
reactor.run()  # the script will block here until the last crawl call is finished
</code></pre>

<p>Afterwards use pycharm as follows</p>

<ul>
<li>connect your scrapy python interpreter to your pycharm project</li>
<li>set <code>main.py</code> as the start script</li>
<li>add a breakpoint in <code>parse_item</code></li>
<li>press/run debug </li>
</ul>

<p>Hopefully this will fix your problem.</p>

python scrapy