Scrapy not executing in the correct order
<p>I am currently working on a web-crawler that is supposed to visit a list of websites in a directory, visit the sites' CSS stylesheets, check for an @media tag (a basic way of checking for responsive design, I know there are other corner cases to consider), and print all websites that do not use responsive design to a file.</p>

<p>I am fairly certain that my method of actually checking the CSS for an @media tag works fine, but the spider is not visiting all the CSS files before deciding whether or not it has found one with an @media tag. I have a test file that logs debugging output as the program progresses, and it shows odd patterns such as finishing checking all CSS files and then printing out what it found in the files, which shouldn't happen.</p>

<p>I was hoping someone could look at my code and help me determine why this isn't happening in the order I want it to.  For reference, the goal is:</p>

<ol>
<li>Visit a website from the list</li>
<li>Visit every CSS file in the head element of that site's HTML</li>
<li>If an @media tag is found, we're done and the site uses responsive design</li>
<li>If not, continue checking more CSS files</li>
<li>If no CSS file contains an @media tag, the site does not use responsive design and should be added to the list</li>
</ol>

<p>Here's my code (not everything works perfectly - for example, the program times out because I haven't worked out using TimeOutError yet, but for the most part, I feel like this should do it's job of correctly evaluating websites, and it is not doing that):</p>

<pre><code>import scrapy
import re
import os.path
from scrapy.linkextractors import LinkExtractor
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from twisted.internet.error import TimeoutError
import time

class LCCISpider(CrawlSpider):
    name = "lcci"
    start_urls = ["http://www.lancasterchamber.com/busdirectory.aspx?mode=category"]
    #Calls parse_item for every category link on main page
    rules = (Rule(SgmlLinkExtractor(restrict_xpaths=('//div[@id="catListingResults"]/table/tr')), 
            callback = 'parse_item', follow = True),)
    website_list = []
    found_media = False

    #Called for each category
    def parse_item(self, response):
        #For each site on the page, calls parse_website


        sites = response.xpath('//div[@id="busListingResults"]/table/tr')
        for site in sites:
            urls = site.xpath('.//td/a[4]/@href').extract()
            for url in urls:
                if len(url) == 0:
                    continue
                else:
                    new_site = response.urljoin(url)
                    yield scrapy.Request(new_site, callback=self.parse_website,
                                                    errback=self.errback_website)




    def parse_website(self, response):

        f = open('output2.txt', 'a')
        f.write("NOW VISITING")
        f.flush()
        f.write(response.url)
        f.flush()
        f.write("\n")
        f.flush()
        f.close()
        #reset found_media to false for each website
        self.found_media = False
        #for every link in the header, check potential css for @media tag
        for href in response.css("head &gt; link::attr('href')"):
            url = response.urljoin(href.extract())
            #if @media tag has not been found, continue checking css
            if self.found_media == False:
                #Call check_css for the url of the css file
                yield scrapy.Request(url, callback=self.check_css,
                                          errback=self.errback_website)

                f = open('output2.txt', 'a')
                f.write("step\n")
                f.flush()
                f.close()
            else:
                break

        #if no @media tag is found in any link in the header, add the url to the website_list

        if self.found_media == False:
            #self.website_list.append(response.url)
            f = open('output2.txt', 'a')
            f.write("No @media tag in")
            f.flush()
            f.write(response.url)
            f.flush()
            f.write("\n")
            f.flush()
            f.close()

            f = open('outputfalse2.txt', 'a')
            f.write(response.url)
            f.write("\n")
            f.close()

        else:
            f = open('outputtrue.txt', 'a')
            f.write(reponse.url)
            f.write("\n")
            f.close()

    def check_css(self, response):

        #Just a way of converting url into a string, the ".txt" is otherwise meaningless
        string = str(response.url)
        f = open('output2.txt', 'a')
        f.write("Checking CSS in ")
        f.write(response.url)
        f.write("\n")
        f.flush()
        f.close()
        #only perform regex search if it's a .css file
        if (string[-4:] == ".css"): 
            media_match = re.search(r'@media', response.body, flags=0)
            if media_match != None:
                f = open('output2.txt', 'a')
                f.write("found @media tag in " + response.url + "\n")
                f.flush()
                #If an @media tag is found, set found_media to True
                self.found_media = True
                f.close()
        else:
            f = open('output2.txt', 'a')
            f.write("not css")
            f.flush()
            f.close()

    def errback_website(self, failure):
        if failure.check(TimeoutError):
            request = failure.request
            self.logger.error = ('TimeoutError on %s', request.url)
</code></pre>

<p>I skimed through and couldn't help but make this work. This is fully cleaned up code.
Very little has changed in terms of logic. 
What it does right now is:</p>

<ol>
<li>Connect to the website</li>
<li>Get all categories</li>
<li>Get all websites from categories</li>
<li>Connect to first page of every website</li>
<li>Look for <code>.css</code> links</li>
<li>Connect to <code>.css</code> links
6.1 If <code>media</code> regex matches yield item with css url and item</li>
</ol>

<p>The only problem here is because of asynchronious nature of scrapy you end up with lots of duplicates since you might crawl multiple .css files at the time. For that we can use simple pipeline to detect and drop duplicates. 
For future reference you shouldn't debug with file writes. Take a look at scrapy shell, you can even use it inside of <code>parse</code> to open up shell during the crawl like:</p>

<pre><code>def parse(self, response):
    inspect_response(response, self)
</code></pre>

<p>Here's the working spider:</p>

<pre><code>import re
from scrapy.spiders import CrawlSpider, Rule
from scrapy.exceptions import DropItem
from scrapy.linkextractors import LinkExtractor
from twisted.internet.error import TimeoutError
from scrapy import Request


class DupePipeline(object):
    def __init__(self):
        self.known_websites = set()

    def process_item(self, item, spider):
        if item['website'] in self.known_websites:
            raise DropItem('duplicate')
        self.known_websites.add(item['website'])
        return item


class LCCISpider(CrawlSpider):
    name = "lcci"
    start_urls = ["http://www.lancasterchamber.com/busdirectory.aspx?mode=category"]
    custom_settings = {
        'ROBOTSTXT_OBEY': False,
        'ITEM_PIPELINES': {
            'myproject.spiders.spider.DupePipeline': 666,
        }
    }
    # Calls parse_item for every category link on main page
    rules = (Rule(LinkExtractor(restrict_xpaths=['//div[@id="catListingResults"]/table/tr']),
                  callback='parse_item', follow=True),)  # why follow?

    # Called for each category
    def parse_item(self, response):
        # For each site on the page, calls parse_website
        sites = response.xpath('//div[@id="busListingResults"]/table/tr')
        for site in sites:
            urls = site.xpath('.//td/a[4]/@href').extract()
            for url in urls:
                if not url:
                    continue
                new_site = response.urljoin(url)
                yield Request(new_site,
                              callback=self.parse_website,
                              errback=self.errback_website)

    def parse_website(self, response):
        # for every link in the header, check potential css for @media tag
        for href in response.css("head &gt; link::attr('href')").extract():
            if not href.endswith('.css'):  # only css files
                continue
            yield Request(response.urljoin(href),
                          meta={'website': response.url},
                          callback=self.check_css,
                          errback=self.errback_website)

    def check_css(self, response):
        media_match = re.search(r'@media', response.body, flags=0)
        if media_match:
            # return item!
            yield {'url': response.url,
                   'website': response.meta['website']}

    def errback_website(self, failure):
        if failure.check(TimeoutError):
            request = failure.request
            self.logger.error = ('TimeoutError on %s', request.url)
</code></pre>

<p>Results after runing for few minutes with <code>scrapy crawl lcci -o test.json</code> I got this: <a href="http://pastebin.com/raw/kfsTKqUY" rel="nofollow">http://pastebin.com/raw/kfsTKqUY</a></p>

python css web scrapy web-crawler