Scrapy Json Rule SgmlLink Extractor
<p>I just want to know on how can I make rule when the website sends me a json response instead of html? On the start url first response, it gives me an html response, but when I navigated through pages, it gives me json response. Here my rule:</p>

<pre><code> Rule(SgmlLinkExtractor(restrict_xpaths=('//div[@class="GridTimeline-items"]'), tags=('div'),
                                   attrs=('data-min-position'), allow=(r''), process_value=my_process_value_friends),
                                    callback='parse_friends', follow=True),
</code></pre>

<p>My question is, how can I apply xpath on json response? </p>

<p>Thank you ,</p>

<p>You can't parse json with xpath or css selectors. You can however turn the json to python dictionary:</p>

<pre><code>import json
def parse(self, response):
    data = json.loads(response.body)
    # then just parse it, e.g.
    item = dict()
    item['name'] = data['name']
    # ...
</code></pre>

<p>Or you can conver json to xml and then parse it with scrapy selectors. There a lot of packages that do that but I'll highlight <a href="https://github.com/quandyfactory/dicttoxml" rel="nofollow"><code>dicttoxml</code></a> in my example:</p>

<pre><code>import json
from dicttoxml import dicttoxml
from scrapy import Selector
def parse(self, response):
    data = json.loads(response.body)
    data_xml = dicttoxml(data)
    sel = Selector(root=data_xml)
    # then parse it
    item = dict()
    item['name'] = sel.xpath("//name/text()")
    # ...
</code></pre>

python json scrapy web-crawler sgml