TypeError: cannot concatenate 'str' and 'NoneType' objects when placing the custom url in scrapy.Request()
<p>I get a url that cannot be used to fetch data from next page, so created a <code>base_url = 'http://www.marinetraffic.com'</code> variable and passed it scrapy request. <code>port_homepage_url = base_url +  port_homepage_url</code>.  It works fine, when i yeild the result like this. <code>yield {'a': port_homepage_url, 'b':item['port_name']}</code> 
I get this result i wanted. </p>

<p><a href="http://www.marinetraffic.com/en/ais/index/ships/range/port_id:20585/port_name:FUJAIRAH%20ANCH,FUJAIRAH" rel="nofollow">http://www.marinetraffic.com/en/ais/index/ships/range/port_id:20585/port_name:FUJAIRAH%20ANCH,FUJAIRAH</a> ANCH</p>

<p>however if place it in scrapy request <code>yield scrapy.Request(port_homepage_url, callback=self.parse, meta={'item': item})</code> i get error  </p>

<pre><code>port_homepage_url = base_url +  port_homepage_url
TypeError: cannot concatenate 'str' and 'NoneType' objects
</code></pre>

<p>here is code</p>

<pre><code>class GetVessel(scrapy.Spider):
    name = "getvessel"
    allowed_domains = ["marinetraffic.com"]
    start_urls = [
        'http://www.marinetraffic.com/en/ais/index/ports/all/flag:AE',
    ]


    def parse(self, response):
        item = VesseltrackerItem()
        base_url = 'http://www.marinetraffic.com'
        for ports in response.xpath('//table/tr[position()&gt;1]'):
            item['port_name'] = ports.xpath('td[2]/a/text()').extract_first()
            port_homepage_url = ports.xpath('td[7]/a/@href').extract_first()
            port_homepage_url = base_url +  port_homepage_url
            yield scrapy.Request(port_homepage_url, callback=self.parse, meta={'item': item})
</code></pre>

<p>The problem does not happen on the initial start URL page, but happens later on when subsequent requests are processed. Take for example <a href="http://www.marinetraffic.com/en/ais/index/ships/range/port_id:1699/port_name:HAMRIYA" rel="nofollow">this page</a>. There are no links in the 7-th <code>td</code> element and, hence, <code>ports.xpath('td[7]/a/@href').extract_first()</code> returns <code>None</code> which results in a failure on the <code>port_homepage_url = base_url +  port_homepage_url</code> line.</p>

<p>How to approach the problem depends on what were you planning to do on the "port" pages. From what I understand, you did not mean to actually handle the "port" page requests with <code>self.parse</code> and need to have a separate callback with different logic inside.</p>

python scrapy