Why Scrapy returns an Iframe?
<p>i want to crawl <a href="http://www.ooshop.com/courses-en-ligne/Home.aspx" rel="nofollow">this site</a> by Python-Scrapy</p>

<p>i try this</p>

<pre><code>class Parik(scrapy.Spider):
    name = "ooshop"
    allowed_domains = ["http://www.ooshop.com/courses-en-ligne/Home.aspx"]

    def __init__(self, idcrawl=None, proxy=None, *args, **kwargs):
        super(Parik, self).__init__(*args, **kwargs)
        self.start_urls = ['http://www.ooshop.com/courses-en-ligne/Home.aspx']

    def parse(self, response):
        print response.css('body').extract_first()
</code></pre>

<p>but i don't have the first page, i have an empty iframe</p>

<pre><code>2016-09-06 19:09:24 [scrapy] DEBUG: Crawled (200) &lt;GET http://www.ooshop.com/courses-en-ligne/Home.aspx&gt; (referer: None)
&lt;body&gt;
&lt;iframe style="display:none;visibility:hidden;" src="//content.incapsula.com/jsTest.html" id="gaIframe"&gt;&lt;/iframe&gt;
&lt;/body&gt;
2016-09-06 19:09:24 [scrapy] INFO: Closing spider (finished)
</code></pre>

<p>The website is protected by Incapsula, a website security service. It's providing your "browser" with a challenge that it must perform before being given a special cookie that gives you access to the website itself.</p>

<p>Fortunately, it's not that hard to bypass. Install <a href="https://github.com/ziplokk1/incapsula-cracker" rel="nofollow">incapsula-cracker</a> and install its downloader middleware:</p>

<pre><code>DOWNLOADER_MIDDLEWARES = {
    'incapsula.IncapsulaMiddleware': 900
}
</code></pre>

python iframe web-scraping scrapy web-crawler