When trying to log in using requests module, I get an error because the User/Password fields are blank
<p>I've been trying to make a scraper to get my grades from my schools website. Unfortunately i cannot log in. When i try to run the program, the return page validates the user/password fields, and since they are blank, it's not letting me proceed.</p>

<p>Also, i am not really sure if I am even coding this correctly.</p>

<pre><code>from twill.commands import *
import requests

payload = {
'ctl00$cphMainContent$lgn$UserName':'user',
'ctl00$cphMainContent$lgn$Password':'pass',
}

cookie = {
'En_oneTime_ga_tracking_v2' : 'true',
'ASP.NET_SessionId' : ''
}

with requests.Session() as s:
    p = s.post('schoolUrl', data=payload, cookies=cookie)
    print p.text
</code></pre>

<hr>

<p>Updated payload:</p>

<pre><code>payload = {
    'ctl00$cphMainContent$lgnEaglesNest$UserName':'user',
    'ctl00$cphMainContent$lgnEaglesNest$Password':'pass',
    '__LASTFOCUS': '',
    '__EVENTTARGET':'',
    '__EVENTARGUMENT':'',
    '__VIEWSTATE':  'LONG NUMBER',
    '__VIEWSTATEGENERATOR': 'C2EE9ABB',
    '__EVENTVALIDATION' : 'LONG NUMBER',
    'ctl00$cphMainContent$lgnEaglesNest$RememberMe': 'on',
    'ctl00$cphMainContent$lgnEaglesNest$LoginButton':'Log+In' 
}
</code></pre>

<p>How do i know if my POST was successful?</p>

<p>The returned page was saying that Username/Password cannot be blank.</p>

<hr>

<p>Complete source:</p>

<pre><code>from twill.commands import *
import requests
payload = {
    'ctl00$cphMainContent$lgnEaglesNest$UserName':'user',
    'ctl00$cphMainContent$lgnEaglesNest$Password':'pass',
    '__LASTFOCUS': '',
    '__EVENTTARGET':'',
    '__EVENTARGUMENT':'',
    '__VIEWSTATE':  'LONG NUMBER',
    '__VIEWSTATEGENERATOR': 'C2EE9ABB',
    '__EVENTVALIDATION' : 'LONG NUMBER',
    'ctl00$cphMainContent$lgnEaglesNest$RememberMe': 'on',
    'ctl00$cphMainContent$lgnEaglesNest$LoginButton':'Log In' 
}

 cookie = {
    'En_oneTime_ga_tracking_v2' : 'true',
    'ASP.NET_SessionId' : ''
 }

with requests.Session() as s:

    loginUrl = 'http://eaglesnest.pcci.edu/Login.aspx?ReturnUrl=%2f'
    gradeUrl = 'http://eaglesnest.pcci.edu/StudentServices/ClassGrades/Default.aspx'
    p = s.post( loginUrl, data=payload)
    print p.text
</code></pre>

<p>First, make sure you are setting the cookies that you need to set correctly. I'd recommend setting them on the session-object:</p>

<pre><code>s = requests.Session()
s.cookies.set("key", "value", domain=".site.com") # the domain is optional
</code></pre>

<p>There are a couple of ways to inspect the results of your POST-request.
Just a few suggestions here:</p>

<pre><code>s = requests.Session()
p = s.post('schoolUrl', data=payload)
print p.text
print p.status_code
print p.cookies
print p.body
print p.headers
</code></pre>

<p>A fact quickly missed is that the <code>requests</code> library silently follows redirects. The response-object will then only contain the last response from the redirect-chain. You can inspect previous responses thusly: <code>p.history</code>. Each element in the returned array is a <code>response</code>-object which you can inspect individually as described above.</p>

<p>Please also unsure that you are fully replicating the request-payload which you can see in google chrome's dev-tools (the network-tab) or the firefox equivalent.</p>

<p>You couldn't per chance post the URL of the site you're trying to login to?</p>

python web-scraping python-requests