Scrapy Request url must be str or unicode, got NoneType:
<p>I try to create my first spider scrapper using scrapy
I use Dmoz as test, I get an error message: 
TypeError: Request url must be str or unicode, got NoneType
But in the Debug I can see the right url</p>

<p>Code:</p>

<pre><code>import scrapy
import urlparse


class DmozSpider(scrapy.Spider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = ["http://www.dmoz.org/search?q=france&amp;all=no&amp;t=regional&amp;cat=all"]

    def parse(self, response):
        sites = response.css('#site-list-content &gt; div.site-item &gt; div.title-and-desc')

        for site in sites:
            yield {
                'name': site.css('a &gt; div.site-title::text').extract_first().strip(),
                'url': site.xpath('a/@href').extract_first().strip(),
                'description': site.css('div.site-descr::text').extract_first().strip(),
            }

        nxt = response.css('#subcategories-div &gt; div.previous-next &gt; div.next-page')
        next_page = nxt.css('a::attr(href)').extract_first()
        if next_page is not None:
            next_page = response.urljoin(next_page)          

        yield scrapy.Request(next_page, callback=self.parse)
</code></pre>

<p>Logs:</p>

<pre><code>2016-10-18 11:17:03 [scrapy] DEBUG: Crawled (200) &lt;GET http://www.dmoz.org/search?q=france&amp;start=20&amp;type=next&amp;all=no&amp;t=regional&amp;cat=all&gt; (referer: http://www.dmoz.org/search?q=france&amp;all=no&amp;t=regional&amp;cat=all)
2016-10-18 11:17:03 [scrapy] ERROR: Spider error processing &lt;GET http://www.dmoz.org/search?q=france&amp;start=20&amp;type=next&amp;all=no&amp;t=regional&amp;cat=all&gt; (referer: http://www.dmoz.org/search?q=france&amp;all=no&amp;t=regional&amp;cat=all)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/defer.py", line 102, in iter_errback
    yield next(it)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
    for x in result:
  File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/referer.py", line 22, in &lt;genexpr&gt;
    return (_set_referer(r) for r in result or ())
  File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/urllength.py", line 37, in &lt;genexpr&gt;
    return (r for r in result or () if _filter(r))
  File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/depth.py", line 58, in &lt;genexpr&gt;
    return (r for r in result or () if _filter(r))
  File "/ENV/bin/tutorial/dirbot/spiders/dmoz.py", line 25, in parse
    yield scrapy.Request(next_page, callback=self.parse)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/http/request/__init__.py", line 25, in __init__
    self._set_url(url)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/http/request/__init__.py", line 51, in _set_url
    raise TypeError('Request url must be str or unicode, got %s:' % type(url).__name__)
TypeError: Request url must be str or unicode, got NoneType:
2016-10-18 11:17:03 [scrapy] INFO: Closing spider (finished)
2016-10-18 11:17:03 [scrapy] INFO: Stored json feed (20 items) in: test.json
2016-10-18 11:17:03 [scrapy] INFO: Dumping Scrapy stats:
</code></pre>

<p>The error is in your code here:</p>

<pre><code>if next_page is not None:
    next_page = response.urljoin(next_page)          

yield scrapy.Request(next_page, callback=self.parse)
</code></pre>

<p>As Padraic Cunningham mentions in his commit: you <code>yield</code> the <code>Request</code> regardless of <code>next_page</code> is <code>None</code> or filled with an URL.</p>

<p>You can solve your problem by changing your code to this:</p>

<pre><code>if next_page is not None:
    next_page = response.urljoin(next_page)          
    yield scrapy.Request(next_page, callback=self.parse)
</code></pre>

<p>where you put your <code>yield</code> inside your <code>if</code> block.</p>

<p>By the way you can change your <code>if</code> to the following:</p>

<pre><code>if next_page:
</code></pre>

<p>because of Python's truth.</p>

<p>And because your spider stops working try to debug your application through <strong>scrapy shell</strong> where you can see if your CSS queries return values or not. You can also add an <code>else</code> to the previous <code>if</code> block which logs / prints a statement to the console that no <code>next_page</code> was found so you know that something is wrong with the site or your CSS queries.</p>

python web-scraping scrapy scrapy-spider