Seeking help to improve a crawler
<p>I'm a beginner with <strong>Scrapy/Python</strong>, I developed a crawler that can find <strong>expired domains</strong> and scan each on a <strong>SEO API</strong>.<br />
My crawler work fine but I pretty sure the crawler isn't 100% optimized for the job.<br/><br/>
Could it be possible to have some tricks to improve the Crawler please ?</p>

<p><strong>expired.py :</strong></p>

<pre><code>class HttpbinSpider(CrawlSpider):
    name = "expired"

    rules = (
        Rule(LxmlLinkExtractor(allow=('.com', '.fr', '.net', '.org', '.info', '.casino', '.eu'),
                               deny=('facebook', 'amazon', 'wordpress', 'blogspot', 'free', 'reddit'),
             callback='parse_obj',
             process_request='add_errback',
             follow=True),
    )

    def __init__(self, domains=None, **kwargs):
        self.start_urls = json.loads(domains)
        super(HttpbinSpider, self).__init__()

    def add_errback(self, request):
        return request.replace(errback=self.errback_httpbin)

    def errback_httpbin(self, failure):
        if failure.check(DNSLookupError):
            request = failure.request
            ext = tldextract.extract(request.url)
            domain = ext.registered_domain
            if domain != '':
                domain = domain.replace("%20", "")
                self.check_domain(domain)

    def check_domain(self, domain):
        if self.is_available(domain) == 'AVAILABLE':

            self.logger.info('## Domain Expired : %s', domain)

            url = 'http://api.majestic.com/api/json?app_api_key=API&amp;cmd=GetIndexItemInfo&amp;items=1&amp;item0=' + domain + '&amp;datasource=fresh'
            response = urllib.urlopen(url)
            data = json.loads(response.read())
            response.close()

            TrustFlow = data['DataTables']['Results']['Data'][0]['TrustFlow']
            CitationFlow = data['DataTables']['Results']['Data'][0]['CitationFlow']
            RefDomains = data['DataTables']['Results']['Data'][0]['RefDomains']
            ExtBackLinks = data['DataTables']['Results']['Data'][0]['ExtBackLinks']

            if (RefDomains &gt; 20) and (TrustFlow &gt; 4) and (CitationFlow &gt; 4):
                insert_table(domain, TrustFlow, CitationFlow, RefDomains, ExtBackLinks)

    def is_available(self, domain):
        url = 'https://api.internet.bs/Domain/Check?ApiKey=KEY&amp;Password=PSWD&amp;responseformat=json&amp;domain' + domain
        response = urllib.urlopen(url)
        data = json.loads(response.read())
        response.close()
        return data['status']
</code></pre>

<p>Thank's a lot.</p>

<p>The biggest issue in your code is urllib requests that are blocking whole async scrapy routine. You can easily replace those with scrapy request chain by yielding a <code>scrapy.Request</code>.</p>

<p>Something like this:</p>

<pre><code>def errback_httpbin(self, failure):
    if not failure.check(DNSLookupError):
        return
    request = failure.request
    ext = tldextract.extract(request.url)
    domain = ext.registered_domain
    if domain == '':
        logging.debug('no domain: {}'.format(request.url))
        return
    domain = domain.replace("%20", "")
    url = 'https://api.internet.bs/Domain/Check?ApiKey=KEY&amp;Password=PSWD&amp;responseformat=json&amp;domain=' + domain
    return Request(url, self.parse_checkdomain)

def parse_checkdomain(self, response):
    """check whether domain is available"""
    data = json.loads(response.read())
    if data['status'] == 'AVAILABLE':
        self.logger.info('Domain Expired : {}'.format(data['domain']))
        url = 'http://api.majestic.com/api/json?app_api_key=API&amp;cmd=GetIndexItemInfo&amp;items=1&amp;item0=' + data['domain']+ '&amp;datasource=fresh'
        return Request(url, self.parse_claim)

def parse_claim(self, response):
    """save available domain's details"""
    data = json.loads(response.read())
    # eliminate redundancy
    results = data['DataTables']['Results']['Data'][0]
    # snake case is more pythonic
    trust_flow = results['TrustFlow']
    citation_flow = results['CitationFlow']
    ref_domains = results['RefDomains']
    ext_back_links = results['ExtBackLinks']

    # don't need to wrap everything in ()
    if ref_domains &gt; 20 and trust_flow &gt; 4 and citation_flow &gt; 4:
        insert_table(domain, trust_flow, citation_flow, ref_domains, ext_back_links)
</code></pre>

<p>This way your code is not being blocked and is fully asynchronious. Generally you don't want to use anything but scrapy requests when dealing with http in your scrapy spider.</p>

python python-2.7 web-scraping scrapy