Scraping ajax page with Scrapy?
<p>I'm using Scrapy for scrape data from this page</p>

<blockquote>
  <p><a href="https://www.bricoetloisirs.ch/magasins/gardena" rel="nofollow">https://www.bricoetloisirs.ch/magasins/gardena</a></p>
</blockquote>

<p>Product list appears dynamically.
Find url to get products</p>

<blockquote>
  <p><a href="https://www.bricoetloisirs.ch/coop/ajax/nextPage/(cpgnum=1&amp;layout=7.01-14_180_69_164_182&amp;uiarea=2&amp;carea=%24ROOT&amp;fwrd=frwd0&amp;cpgsize=12)/.do?page=2&amp;_=1473841539272" rel="nofollow">https://www.bricoetloisirs.ch/coop/ajax/nextPage/(cpgnum=1&amp;layout=7.01-14_180_69_164_182&amp;uiarea=2&amp;carea=%24ROOT&amp;fwrd=frwd0&amp;cpgsize=12)/.do?page=2&amp;_=1473841539272</a></p>
</blockquote>

<p>But when i scrape it by Scrapy it give me empty page </p>

<pre><code>&lt;span class="pageSizeInformation" id="page0" data-page="0" data-pagesize="12"&gt;Page: 0 / Size: 12&lt;/span&gt;
</code></pre>

<p>Here is my code </p>

<pre><code># -*- coding: utf-8 -*-
import scrapy

from v4.items import Product


class GardenaCoopBricoLoisirsSpider(scrapy.Spider):
    name = "Gardena_Coop_Brico_Loisirs_py"

    start_urls = [
            'https://www.bricoetloisirs.ch/coop/ajax/nextPage/(cpgnum=1&amp;layout=7.01-14_180_69_164_182&amp;uiarea=2&amp;carea=%24ROOT&amp;fwrd=frwd0&amp;cpgsize=12)/.do?page=2&amp;_=1473841539272'
        ]

    def parse(self, response):
        print response.body
</code></pre>

<p>I believe you need to send an additional request just like a browser does. Try to modify your code as follows:</p>

<pre><code># -*- coding: utf-8 -*-
import scrapy

from scrapy.http import Request
from v4.items import Product


class GardenaCoopBricoLoisirsSpider(scrapy.Spider):
    name = "Gardena_Coop_Brico_Loisirs_py"

    start_urls = [
        'https://www.bricoetloisirs.ch/coop/ajax/nextPage/'
    ]

    def parse(self, response):
        request_body = '(cpgnum=1&amp;layout=7.01-14_180_69_164_182&amp;uiarea=2&amp;carea=%24ROOT&amp;fwrd=frwd0&amp;cpgsize=12)/.do?page=2&amp;_=1473841539272'
        yield Request(url=response.url, body=request_body, callback=self.parse_page)

    def parse_page(self, response):
        print response.body
</code></pre>

python ajax scrapy