Extract data from login authentication website using scrapy
<p>I am trying to login first and then extract data from pages which are visible after login. my spider is- </p>

<pre><code>import scrapy
from scrapy.selector import HtmlXPathSelector
from scrapy.http.request import Request
from scrapy.spiders import BaseSpider
from scrapy.http import FormRequest
from loginform import fill_login_form

class ElementSpider(scrapy.Spider):
    name = 'example'
    start_urls = ['https://github.com/login']

    def parse(self, response):
        return [FormRequest.from_response(response,
                    formdata={'login': 'myid', 'password': 'my password'},
                    callback=self.after_login)]

    def after_login(self, response):

        if "Incorrect username or password" in response.body:
            print "hey"
            self.log("Login failed", level=log.ERROR)
            return

        else:
            return Request(url="https://github.com/settings/emails",
                   callback=self.parse_data)


    def parse_data(self, response):
        email = response.xpath('//div[@class="boxed-group-inner"]/li[@class="clearfix css-truncate settings-email"]/span[@class="css-truncate-target"]/text()').extract()
        print email
</code></pre>

<p>I am getting nothing in the output.
Is there error in implementation ???</p>

<p>You haven't created an instance of your class <code>ElementSpider</code>.<br>
You first need to create an instance of the class.<br><br>
<strong>NOTICE</strong><br> Every class should have a constructor, Therefor it is recommended you should implement the <code>__init__</code> method in your class.<br></p>

<p>This is how the code is should to look.</p>

<pre><code>import scrapy
from scrapy.selector import HtmlXPathSelector
from scrapy.http.request import Request
from scrapy.spiders import BaseSpider
from scrapy.http import FormRequest
from loginform import fill_login_form

class ElementSpider(scrapy.Spider):
    name = 'example'
    start_urls = ['https://github.com/login']

    def __init__(self, *args, **kwargs):
                super(ElementSpider, self).__init__(*args, **kwargs)

    def parse(self, response):
        return [FormRequest.from_response(response,
                    formdata={'login': 'myid', 'password': 'my password'},
                    callback=self.after_login)]

    def after_login(self, response):

        if "Incorrect username or password" in response.body:
            print "hey"
            self.log("Login failed", level=log.ERROR)
            return

        else:
            return Request(url="https://github.com/settings/emails",
                   callback=self.parse_data)


    def parse_data(self, response):
        email = response.xpath('//*[@id="settings-emails"]/li/span[@class="css-truncate-target"]').extract()
        print email


if __name__ == "__main__":
    spider = ElementSpider()
</code></pre>

python scrapy web-crawler