Dynamically setting scrapy request call back
<p>I'm working with scrapy.  I want to rotate proxies on a per request basis and get a proxy from an api I have that returns a single proxy. My plan is to make a request to the api, get a proxy,  then use it to set the proxy based on :</p>

<pre><code>http://stackoverflow.com/questions/39430454/making-request-to-api-from-within-scrapy-function
</code></pre>

<p>I have the following:</p>

<pre><code>class ContactSpider(Spider):
    name = "contact"

def parse(self, response):

....
        PR = Request(
        'my_api'
        headers=self.headers,
        meta={'newrequest': Request(url_to_scrape,  headers=self.headers),},
        callback=self.parse_PR
    )
    yield PR


def parse_PR(self, response):
    newrequest = response.meta['newrequest']
    proxy_data = response.body
    newrequest.meta['proxy'] = 'http://'+proxy_data
    newrequest.replace(url = 'http://ipinfo.io/ip') #TESTING
    newrequest.replace(callback= self.form_output) #TESTING

    yield newrequest

def form_output(self, response):
    open_in_browser(response)
</code></pre>

<p>but I'm getting:</p>

<pre><code>    Traceback (most recent call last):
  File "C:\twisted\internet\defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "C:\twisted\python\failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "C:\scrapy\core\downloader\middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
  File "C:\scrapy\utils\defer.py", line 45, in mustbe_deferred
    result = f(*args, **kw)
  File "C:\scrapy\core\downloader\handlers\__init__.py", line 65, in download_request
    return handler.download_request(request, spider)
  File "C:\scrapy\core\downloader\handlers\http11.py", line 60, in download_request
    return agent.download_request(request)
  File "C:\scrapy\core\downloader\handlers\http11.py", line 255, in download_request
    agent = self._get_agent(request, timeout)
  File "C:\scrapy\core\downloader\handlers\http11.py", line 235, in _get_agent
    _, _, proxyHost, proxyPort, proxyParams = _parse(proxy)
  File "C:\scrapy\core\downloader\webclient.py", line 37, in _parse
    return _parsed_url_args(parsed)
  File "C:\scrapy\core\downloader\webclient.py", line 20, in _parsed_url_args
    host = b(parsed.hostname)
  File "C:\scrapy\core\downloader\webclient.py", line 17, in &lt;lambda&gt;
    b = lambda s: to_bytes(s, encoding='ascii')
  File "C:\scrapy\utils\python.py", line 117, in to_bytes
    'object, got %s' % type(text).__name__)
TypeError: to_bytes must receive a unicode, str or bytes object, got NoneType
</code></pre>

<p>what am I doing wrong?</p>

<p>The stacktrace info suggests Scrapy has encountered a request object whose <code>url</code> is <code>None</code>, which is expected to be of string type.</p>

<p>These two lines in your code:</p>

<pre><code>newrequest.replace(url = 'http://ipinfo.io/ip') #TESTING
newrequest.replace(callback= self.form_output) #TESTING
</code></pre>

<p>would not work as expected, since method <a href="http://doc.scrapy.org/en/latest/topics/request-response.html#scrapy.http.Request.replace" rel="nofollow"><code>Request.replace</code></a> <em>returns a new instance</em> instead of modifying the original request in-place.</p>

<p>You would need something like this:</p>

<pre><code>newrequest = newrequest.replace(url = 'http://ipinfo.io/ip') #TESTING
newrequest = newrequest.replace(callback= self.form_output) #TESTING
</code></pre>

<p>or simply:</p>

<pre><code>newrequest = newrequest.replace(
    url='http://ipinfo.io/ip',
    callback=self.form_output
)
</code></pre>

python scrapy