Return 400 when login to a website using python requests
<p>I am trying to write a python script to login to a website using the requests library.
This is the login form.</p>

<pre><code>&lt;form action="/login" method="POST"&gt;&lt;input type="hidden" name="post_key" value="b762c617d52cf987fdb40d74c6a04e07"&gt;&lt;input type="hidden" name="return_to" value="http://www.pixiv.net/"&gt;&lt;input type="hidden" name="lang" value="en"&gt;&lt;input type="hidden" name="source" value="pc"&gt;&lt;div class="input-field-group"&gt;&lt;div class="input-field"&gt;&lt;input type="text" name="pixiv_id" placeholder="E-mail address / pixiv ID" autocapitalize="off"&gt;&lt;/div&gt;&lt;div class="input-field"&gt;&lt;input type="password" name="password" placeholder="Password" autocapitalize="off"&gt;
</code></pre>

<p>This is my code.</p>

<pre><code>import requests

url = "https://accounts.pixiv.net/login"

# set requests headers
headers = {
'Connection':'keep-alive',
'User-agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36',
'Content-Type':'application/x-www-form-urlencoded; charset=UTF-8'
}

# get user id and password
pixiv_id = raw_input("Your pixiv id : ")
password = raw_input("Your pixiv password: ")

payload = {
    'action' : '/login',
    'return_to' : 'http://www.pixiv.net'
}
payload['pixiv_id']=pixiv_id
payload['password']=password


with requests.Session() as s:
    r = s.post(url, data=payload, headers=headers)
    response = s.get("http://www.pixiv.net")
    print r.status_code
    print response.text
</code></pre>

<p>My question is, should I fill in all the hidden value in the form?
Also, I have run it for many times however it always return 400. Could anyone help me figuring out the problem of my code? </p>

<p>When I log in and look into browser developer tools I see much more POST request parameters being sent after clicking "log in":</p>

<p><a href="http://i.stack.imgur.com/dmaNZm.png" rel="nofollow"><img src="http://i.stack.imgur.com/dmaNZm.png" alt="enter image description here"></a> </p>

<p><code>requests</code> would send only what you explicitly tell it to send - meaning, you should also send all the hidden form parameters. It might involve HTML parsing - you can use, for example, <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/" rel="nofollow"><code>BeautifulSoup</code></a> for it.</p>

<p>Or, you can use tools like <a href="http://wwwsearch.sourceforge.net/mechanize/" rel="nofollow"><code>mechanize</code></a>, <a href="https://github.com/hickford/MechanicalSoup" rel="nofollow"><code>mechanicalsoup</code></a> or <a href="https://github.com/jmcarp/robobrowser" rel="nofollow"><code>robobrowser</code></a> which would auto-discover and send the hidden attributes of a form.</p>

python http login python-requests