How can I search and parse multiple, session-specific search results with Scrapy?
<p>[EDIT: Code is revised to show my attempt at doing Granitosaurus' suggestion]</p>

<p>I need to execute some predefined searches and crawl the results 1 level deep, then parse those pages. This has to be done sequentially because the site will not allow results to be crawled unless those results were just retrieved pursuant to a search.</p>

<p>My predefined searches are in a list and I'm trying to loop through that list, but I cannot get it to work. Scrapy either jumps all around or loops through all the start_requests without parsing each as it goes. Can anyone show me a way to do this? Thanks!</p>

<p>The search terms are simply pairs of date ranges which I've stored in a list at the beginning.</p>

<p>Here's my [NEWLY EDITED] code with my non-working loop commented out:</p>

<pre><code>import scrapy


monthlist = [{'from_date': '9/1/13', 'to_date': '9/30/13'},
    {'from_date': '10/1/13', 'to_date': '10/31/13'},
    {'from_date': '11/1/13', 'to_date': '11/30/13'},
    {'from_date': '12/1/13', 'to_date': '12/31/13'},
    {'from_date': '1/1/14', 'to_date': '1/31/14'},
    {'from_date': '2/1/14', 'to_date': '2/28/14'},
    {'from_date': '3/1/14', 'to_date': '3/31/14'},
    {'from_date': '4/1/14', 'to_date': '4/30/14'},
    {'from_date': '5/1/14', 'to_date': '5/31/14'},
    {'from_date': '6/1/14', 'to_date': '6/30/14'},
    {'from_date': '7/1/14', 'to_date': '7/31/14'},
    {'from_date': '8/1/14', 'to_date': '8/31/14'},
    {'from_date': '9/1/14', 'to_date': '9/30/14'}]
    # remainder of list snipped for this post


class docketSpider(scrapy.Spider):
    name = 'robey'
    custom_settings = {
        'DEPTH_PRIORITY': 1,
        'SCHEDULER_DISK_QUEUE': 'scrapy.squeues.PickleFifoDiskQueue',
        'SCHEDULER_MEMORY_QUEUE': 'scrapy.squeues.FifoMemoryQueue'
    }
    start_urls = ['http://justice1.dentoncounty.com/PublicAccessDC/default.aspx']
    monkeycount = 0
    def start_requests(self):
        for i in range(33):
            self.monkeycount =+ 1
            yield scrapy.Request('http://justice1.dentoncounty.com/PublicAccessDC/default.aspx', dont_filter=True)
            yield scrapy.Request('http://justice1.dentoncounty.com/PublicAccessDC/default.aspx', self.parse, dont_filter=True)
        # yield scrapy.Request('http://justice1.dentoncounty.com/PublicAccessDC/Search.aspx?ID=200&amp;NodeID=1256&amp;NodeDesc=393rd%20Judicial%20District%20Court', dont_filter=True)

    def parse(self, response):
        request = scrapy.Request('http://justice1.dentoncounty.com/PublicAccessDC/Search.aspx?ID=200&amp;NodeID=1256&amp;NodeDesc=393rd%20Judicial%20District%20Court',
            callback=self.retrieve_caselist)
        return request

    def retrieve_caselist(self, response):
        # caserange = {'from_date': '9/1/2013', 'to_date': '9/30/2013'}
        # for caserange in monthlist:
        yield scrapy.FormRequest.from_response(
            response,
            url='http://justice1.dentoncounty.com/PublicAccessDC/Search.aspx?ID=200&amp;NodeID=1256&amp;NodeDesc=393rd%20Judicial%20District%20Court',
            formdata={'CaseSearchMode': 'CaseNumber', 'SearchBy': '6', 'ExactName': 'on', 'PartySearchMode': 'Name', 'AttorneySearchMode': 'Name', 'cboState': 'AA', 'CaseStatusType': '0', 'SortBy': 'fileddate', 'SearchSubmit': 'Search', 'SearchType': 'CASE', 'StatusType': 'true', 'AllStatusTypes': 'true', 'SearchParams': 'DateFiled~~Search By:~~6~~Date Filed||chkExactName~~Exact Name:~~on~~on||AllOption~~Case Status:~~0~~All||DateFiledOnAfter~~Date Filed On or After:~~' + monthlist[self.monkeycount]['from_date'] + '~~' + monthlist[self.monkeycount]['from_date'] + '||DateFiledOnBefore~~Date Filed On or Before:~~' + monthlist[self.monkeycount]['to_date'] + '~~' + monthlist[self.monkeycount]['to_date'] + '||selectSortBy~~Sort By:~~Filed Date~~Filed Date', 'SearchMode': 'FILED', 'DateFiledOnAfter': monthlist[self.monkeycount]['from_date'], 'DateFiledOnBefore': monthlist[self.monkeycount]['to_date']},
            callback=self.parse_caselist,
            dont_filter=True
        )

    def parse_caselist(self, response):
        for href in response.xpath('//html/body/table/tr/td/a/@href').extract():
            full_url = response.urljoin(href)
            yield scrapy.Request(full_url, callback=self.parse_casedetail)


    def parse_casedetail(self, response):
        yield {
            'case_num': response.xpath('/html/body/div[2]/span/text()').extract(),
            'attorneys': response.xpath('/html/body/table[4]/tr/td/b/text()').extract(),
            'file_date': response.xpath('/html/body/table[3]/tr/td[3]/table/tr/td/table/tr[2]/td/b/text()').extract(),
            'case_type': response.xpath('/html/body/table[3]/tr/td[3]/table/tr/td/table/tr[1]/td/b/text()').extract()
        }
</code></pre>

<p>When this runs, it loops through the loop making each request, but it fails to call self.parse on each loop (along with all the callables that run in turn, after self.parse). After the loop, it parses a single set of search results, then acts like it's finished. Here's the <a href="http://pastebin.com/embed/7Bn0DM3U" rel="nofollow">pastebin log of it running</a> from PyCharm. Any ideas?  </p>

<p>Seems like you are being restricted by the website session. What you could try is to spawn multiple sessions in <code>start_requests()</code> and then for each session yield unique search query.</p>

<p>Something like: </p>

<pre><code>class MySpider(scrapy.Spider):
    name = "myspider"
    start_url = "http://scrapy.org"
    search_options = [
        {'name': 'john', 'last_name': 'snow'},
        {'name': 'james', 'last_name': 'bond'},
    ]

    def start_requests(self):
        # Start a web session for every search option
        for i, option in enumerate(self.search_options):
            yield Request(self.start_url,
                          dont_filter=True,
                          meta={'search_option': option,
                                'cookiejar': i})

    def parse(self, response):
        # make search request with those specific options!
        yield FormRequest.from_response(response, 
                                        callback=self.parse_search,
                                        meta=response.meta,
                                        formdata=response['search_option'])

    def parse_search(self, response):
        # parse your search here
</code></pre>

python asp.net scrapy