How to simulate the process of login using scrapy, when the information might be tranferred by ajax?
<p>I'm doing some crawling jobs on <a href="http://www.asianmetal.cn/" rel="nofollow">http://www.asianmetal.cn/</a>.
But I fail to login using a simple FormRequest like this:</p>

<pre><code>def start_requests(self):
    url = 'http://www.asianmetal.cn/login/ajaxLogin.am'
    fake_header = httputils.random_headers()
    formdata = {
        'txtUser_LoginName': '***',
        'txtUser_Pwd': '***',
        'tfc': '1',
    }
    yield scrapy.FormRequest(url=url, formdata=formdata,
        headers=fake_header, callback=self.parse)
</code></pre>

<p>However it doesn't work. So I did screenshots to find out what happened.</p>

<p><img src="http://i.stack.imgur.com/U7giI.jpg" alt="Formdata"></p>

<p>So how should I simulate the login process?</p>

<p>You are probably missing some headers. I've tried gibberish log in and these are the headers that were sent:</p>

<p><a href="http://i.stack.imgur.com/tdrkB.png" rel="nofollow"><img src="http://i.stack.imgur.com/tdrkB.png" alt="enter image description here"></a></p>

<p>The only important here is probably <code>X-Requested-With</code>. Some websites also check <code>Referer</code> but that's really rare.</p>

<p>To fix this simply update your fake header with approriate values:</p>

<pre><code>fake_header = {}
fake_header['X-Request-With'] = 'XMLHttpRequest'
fake_header['Referer'] = response.url
</code></pre>

python ajax scrapy