Wait for a Request to complete - Python Scrapy
<p>I have a Scrapy Spider which scrapes a website and that website requires to refresh a token to be able to access them.</p>

<pre><code>def get_ad(self, response):
    temp_dict = AppextItem()
    try:
        Selector(response).xpath('//div[@class="messagebox"]').extract()[0]
        print("Captcha found when scraping ID "+ response.meta['id'] + " LINK: "+response.meta['link'])
        self.p_token = ''

        return Request(url = url_, callback=self.get_p_token, method = "GET",priority=1, meta = response.meta)

    except Exception:
        print("Captcha was not found")
</code></pre>

<p>I have a <code>get_p_token</code> method that refreshes token and assigns to <code>self.p_token</code> </p>

<p><code>get_p_token</code> is called when Captcha is found, but problem is, other Requests keep executing.</p>

<p>I want that if Captcha is found, do not make next request until execution of <code>get_p_token</code> is finished.</p>

<p>I have <code>priority=1</code> but that does not help.</p>

<p><a href="http://pastebin.com/X6Q4ZFp2" rel="nofollow">HERE is full code of Spider</a></p>

<p>P.S:</p>

<p>Actually that token is passed to each URL so that is why I want to wait until a new token is found and then scrape the rest of URLs.</p>

<p>This is how I would go on about it:</p>

<pre><code>def get_p_token(self, response):
    # generate token
    ...
    yield Request(url = response.url, callback=self.no_captcha, method = "GET",priority=1, meta = response.meta, dont_filter=True)


def get_ad(self, response):
    temp_dict = AppextItem()
    try:
        Selector(response).xpath('//div[@class="messagebox"]').extract()[0]
        print("Captcha found when scraping ID "+ response.meta['id'] + " LINK: "+response.meta['link'])
        self.p_token = ''

        yield Request(url = url_, callback=self.get_p_token, method = "GET",priority=1, meta = response.meta)

    except Exception:
        print("Captcha was not found")
        yield Request(url = url_, callback=self.no_captcha, method = "GET",priority=1, meta = response.meta)
</code></pre>

<p>You haven't provided working code so this is only a demonstration of the problem...The logic here is pretty simple:</p>

<p>If a captcha is found it goes to <code>get_p_token</code> and after generating the token, it requests the url that you were requesting before. If no captcha is found it goes on as normal.</p>

python scrapy screen-scraping scrapy-spider