Login in to AJAX form using scrapy
<p>I am trying to logging in to the page <a href="http://www.kumhoepos.com.au/epos/index.jsp" rel="nofollow">http://www.kumhoepos.com.au/epos/index.jsp</a> which is java scripted. I am trying this but not successful.</p>

<pre><code>class ElementSpider(scrapy.Spider):
    name = 'epos'
    start_urls = ['http://www.kumhoepos.com.au/epos/index.jsp']

    def parse(self, response):
        yield scrapy.FormRequest.from_response(response,
                    formdata={'user_id': 'myid', 'pass': 'mypassword'},
                    callback = self.after_login)



    def after_login(self, response):
        if "Password incorrect" in response.body:
            print "hey"
            self.log("Login failed", level=log.ERROR)

        elif "Unregistered ID" in response.body:
            print "hello"
            self.log("Login failed", level=log.ERROR)

        # We've successfully authenticated, let's have some fun!
        else:
            print response.body
            url_pass = 'http://www.kumhoepos.com.au/epos/login.do?action=logincheck'
            yield scrapy.Request(url="http://www.kumhoepos.com.au/epos/manage/sicus.do?action=list",
                    callback=self.parse_1)

    def parse_1(self, response):

        print response.body
        x = response.xpath('//div[@id="contents_wrapper"]/div[@class="contents_top"]/h1/text()').extract()
        print x

        table = response.xpath('//table[@class="tbl_type02"]/tbody/tr//td//text()').extract()
        print table
</code></pre>

<p>And the output is-</p>

<pre><code>D:\xampp\htdocs\py\test1&gt;scrapy crawl epos -L WARNING
&lt;!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.or
g/TR/xhtml1/DTD/xhtml1-transitional.dtd"&gt;

&lt;html&gt;
&lt;body&gt;
&lt;form name='signForm' method='post'&gt;
&lt;input type=hidden name="user_id" /&gt;
&lt;input type=hidden name="pass" /&gt;
&lt;/form&gt;
&lt;script type="text/javascript"&gt;
        document.signForm.action = "/epos/login.do?action=login";
        document.signForm.user_id.value = 'myid';
        document.signForm.pass.value    = 'mypassword';
        document.signForm.target = "_parent";
        document.signForm.submit();
&lt;/script&gt;
&lt;/body&gt;
&lt;/html&gt;


&lt;script language='javascript'&gt;
alert('Session has been terminated. Go to the login page.');
 if(window.opener &amp;&amp; !window.opener.closed )
 {
    opener.parent.location.href = '/epos/';
    self.close();
 }else{
    parent.location.href='/epos/';
 }
&lt;/script&gt;

[]
[]
</code></pre>

<p>Please suggest how to handle java-scripted form. </p>

<p>The Scrapy does not support on-page JS evaluation.
The submit button triggers JS <a href="http://view-source:http://www.kumhoepos.com.au/epos/index.jsp" rel="nofollow">login()</a> function:</p>

<pre><code>function login()
{
    var cookie_user_id ='KHTIRE_USER_ID';

    var f = document.loginForm;
    if(isNull(f.user_id,   'Please input your id.'))  return false;
    if(isNull(f.pass,      'Please input your password.'))  return false;
    if(f.chk_save.checked){
        setCookie(cookie_user_id, f.user_id.value, 5);
    }else{
        deleteCookie(cookie_user_id) ;
    }

    f.target = "indexIfrm";
    f.submit();
}
</code></pre>

<p>Inside of that function cookie is handled by page's JS function <a href="http://view-source:http://www.kumhoepos.com.au/epos/index.jsp" rel="nofollow">setCookie</a>:</p>

<pre><code>function setCookie( cookieName, cookieValue, expireDate )
{
    var today = new Date();
    today.setDate( today.getDate() + parseInt( expireDate ) );
    document.cookie = cookieName + "=" + escape( cookieValue ) + "; path=/; expires=" + today.toGMTString() + ";";
}
</code></pre>

<p>Most likely you should not only imitate a form submission, but also set browser cookies. How? - imitate a cookie jar like <a href="http://scraping.pro/http-cookie-handler-curl/" rel="nofollow">here in php</a>. </p>

<p><code>f.target="indexIfrm"</code> points to the iframe where the submit response will be loaded<a href="http://www.w3schools.com/tags/att_form_target.asp" rel="nofollow"> [docs]</a>. 
So, check server ajax response and extract what you need from JS-stuffed response...
See for example a response that I've got for a wrong {id/password} from that ajax form submission:</p>

<pre><code>&lt;script language='javascript' type=""&gt;
    alert('Unregistered ID. Please apply to the administrator...');
&lt;/script&gt; 

&lt;script type="text/javascript"&gt;    
    parent.loginForm.user_id.focus();    
&lt;/script&gt;
</code></pre>

<p>Read <a href="http://scraping.pro/javascript-protected-content-scrape/" rel="nofollow">more here</a> of how to handle JS-stuffed web pages.</p>

python web-scraping scrapy