Scrapy response have backslashes into element attributes
<p>I run the following code in a Scrapy Shell, to scrape data using a POST request:</p>

<pre><code>url = 'http://www.ldg.co.uk/wp-admin/admin-ajax.php'

data = {'action': 'wpp_property_overview_pagination',
        'wpp_ajax_query[show_children]': 'true',
        'wpp_ajax_query[disable_wrapper]': 'true',
        'wpp_ajax_query[pagination]': 'off',
        'wpp_ajax_query[per_page]': '10',
        'wpp_ajax_query[query][property_category]': 'residential',
        'wpp_ajax_query[query][listing_type]': 'rent',
        'wpp_ajax_query[query][sort_by]': 'price_rent',
        'wpp_ajax_query[query][sort_order]': 'ASC',
        'wpp_ajax_query[query][pagi]': '0--10',
        'wpp_ajax_query[sorter]': '',
        'wpp_ajax_query[sort_by]': 'price_rent',
        'wpp_ajax_query[sort_order]': 'ASC',
        'wpp_ajax_query[template]': 'ajax',
        'wpp_ajax_query[requested_page]': '2'}

request = FormRequest(url, formdata = data)
fetch(request)
</code></pre>

<p>I know that inside the response are elements with the class <code>"property-thumb"</code>, I've checked it by using Chrome Dev Tools, reading the response content. So, I try to scrape data using the XPath <code>//*[@class="property-thumb"]</code>, this XPath is right (I use a Chrome plugin to check it with the content loaded into the page), but it isn't right if I try to use it from the Scrapy Shell:</p>

<pre><code>In [10]: response.xpath('//*[@class="property-thumb"]')
Out[10]: []
</code></pre>

<p>I have noticed that <code>response.body</code> comes with a lot of backslashes, so I've figured out that the correct XPath should be <code>//*[@class=\'\\"property-thumb\\"\']</code>:</p>

<pre><code>In [11]: response.xpath('//*[@class=\'\\"property-thumb\\"\']')
Out[11]: 
[&lt;Selector xpath='//*[@class=\'\\"property-thumb\\"\']' data=u'&lt;div class=\'\\"property-thumb\\"\'&gt;\\n      '&gt;,
 &lt;Selector xpath='//*[@class=\'\\"property-thumb\\"\']' data=u'&lt;div class=\'\\"property-thumb\\"\'&gt;\\n      '&gt;,
 &lt;Selector xpath='//*[@class=\'\\"property-thumb\\"\']' data=u'&lt;div class=\'\\"property-thumb\\"\'&gt;\\n      '&gt;]
</code></pre>

<p>I think there is a problem with the way Scrapy manages strings from responses. Also, I think that those backslashes can generate more problems when scraping. Why do this happen? How can I solve it to use normal XPaths?</p>

<p>There is a very simple solution, you get <a href="http://www.w3schools.com/json/" rel="nofollow"><em>json</em></a> back <em>not html</em>:</p>

<pre><code>url = 'http://www.ldg.co.uk/wp-admin/admin-ajax.php'

data = {'action': 'wpp_property_overview_pagination',
        'wpp_ajax_query[show_children]': 'true',
        'wpp_ajax_query[disable_wrapper]': 'true',
        'wpp_ajax_query[pagination]': 'off',
        'wpp_ajax_query[per_page]': '10',
        'wpp_ajax_query[query][property_category]': 'residential',
        'wpp_ajax_query[query][listing_type]': 'rent',
        'wpp_ajax_query[query][sort_by]': 'price_rent',
        'wpp_ajax_query[query][sort_order]': 'ASC',
        'wpp_ajax_query[query][pagi]': '0--10',
        'wpp_ajax_query[sorter]': '',
        'wpp_ajax_query[sort_by]': 'price_rent',
        'wpp_ajax_query[sort_order]': 'ASC',
        'wpp_ajax_query[template]': 'ajax',
        'wpp_ajax_query[requested_page]': '2'}
import requests
print(requests.post(url, data).json())
</code></pre>

<p>Which would give you:</p>

<pre><code>{u'display': u'        &lt;section class="property-card new-post"&gt;\n            &lt;div class="property-thumb"&gt;\n                &lt;a class="property-image" href="http://www.ldg.co.uk/residential/1-bedroom-property-for-rent-lisson-street-marylebone-london-101588004937/" title="Lisson Street, Marylebone, London"&gt;\n                    &lt;img src="http://www.ldg.co.uk/wp-content/uploads/2016/08/IMG_4427_6_large.jpg" alt="Lisson Street, Marylebone, London thumbnail"&gt;\n\n                                    &lt;/a&gt;\n            &lt;/div&gt;&lt;!-- /.property-thumb --&gt;\n\n            &lt;div class="property-content"&gt;\n                                    &lt;header class="property-title"&gt;\n                        &lt;h2&gt;\n                            &lt;a  href="http://www.ldg.co.uk/residential/1-bedroom-property-for-rent-lisson-street-marylebone-london-101588004937/"&gt;Lisson Street, Marylebone, London&lt;/a&gt;\n                        &lt;/h2&gt;\n                    &lt;/header&gt;\n                \n                &lt;span class="property-style-tenure"&gt;&lt;/span&gt;\n                &lt;div class="property-details"&gt;\n\n                    \n                                                    &lt;div class="property-price"&gt;\n                                &lt;div class="property-style-tenure"&gt;&lt;span&gt;&lt;/span&gt;&lt;/div&gt;\xa3420&lt;small&gt;/pw&lt;/small&gt;\n                                                                    &lt;span class="fees-link-wrapper"&gt;+ &lt;a target="_blank" href="http://www.ldg.co.uk/residential/property-lettings/fees-and-charges/"&gt;fees&lt;/a&gt;&lt;/span&gt;\n                                                            &lt;/div&gt;\n                        \n                    \n                    &lt;div class="property-features"&gt;\n                                                    &lt;div class="property-feature"&gt;\n                                &lt;div class="property-living_rooms"&gt;\n                                    &lt;span class="esf-icon esf-32 esf-icon-living_rooms"&gt;&lt;/span&gt;\n                                    1                                    Reception                                &lt;/div&gt;\n                            &lt;/div&gt;\n                        \n                                                    &lt;div class="property-feature"&gt;\n                                &lt;div class="property-bedrooms"&gt;\n                                    &lt;span class="esf-icon esf-32 esf-icon-bedrooms"&gt;&lt;/span&gt;\n                                    1                                    Bedroom                                &lt;/div&gt;\n                            &lt;/div&gt;\n                        \n                                                    &lt;div class="property-feature"&gt;\n                                &lt;div class="property-bathrooms"&gt;\n                                    &lt;span class="esf-icon esf-32 esf-icon-bathrooms"&gt;&lt;/span&gt;\n                                    1                                    Bathroom                                &lt;/div&gt;\n                            &lt;/div&gt;\n                                            &lt;/div&gt;&lt;!-- /.property-features --&gt;\n\n\n                        &lt;div class="property-media"&gt;\n                                                              &lt;a href="http://www.ldg.co.uk/wp-content/uploads/2016/09/FLP_4427_1_large-743x1024.png" target="_blank" class="alternative-link fancybox " rel="fancybox-group"&gt;View Floor Plan&lt;/a&gt;\n      \n                                                                                                &lt;span class="separator"&gt;|&lt;/span&gt;\n                                                                &lt;a href="http://media2.jupix.co.uk/v3/clients/1588/properties/4427/MED_4427_6235.pdf" target="_blank" class="alternative-link"&gt;Download Brochure&lt;/a&gt;\n                                                    &lt;/div&gt;&lt;!-- /.property-media --&gt;\n\n                    &lt;div class="property-read-more"&gt;\n                        &lt;a href="http://www.ldg.co.uk/residential/1-bedroom-property-for-rent-lisson-street-marylebone-london-101588004937/" class="btn btn-sm lighter-dark-primary-color"&gt;\n                            View Details\n                        &lt;/a&gt;\n                    &lt;/div&gt;\n                &lt;/div&gt;&lt;!-- /.property-details --&gt;\n            &lt;/div&gt;&lt;!-- /.property-content --&gt;\n        &lt;/section&gt;\n            &lt;section class="property-card new-post"&gt;\n            &lt;div class="property-thumb"&gt;\n                &lt;a class="property-image" href="http://www.ldg.co.uk/residential/1-bedroom-property-for-rent-riding-house-street-fitzrovia-london-101588003963/" title="Riding House Street, Fitzrovia, London"&gt;\n                    &lt;img src="http://www.ldg.co.uk/wp-content/uploads/2016/09/IMG_3453_10_large.jpg" alt="Riding House Street, Fitzrovia, London thumbnail"&gt;\n\n                                    &lt;/a&gt;\n            &lt;/div&gt;&lt;!-- /.property-thumb --&gt;\n\n            &lt;div class="property-content"&gt;\n                                    &lt;header class="property-title"&gt;\n                        &lt;h2&gt;\n                            &lt;a  href="http://www.ldg.co.uk/residential/1-bedroom-property-for-rent-riding-house-street-fitzrovia-london-101588003963/"&gt;Riding House Street, Fitzrovia, London&lt;/a&gt;\n                        &lt;/h2&gt;\n                    &lt;/header&gt;\n                \n                &lt;span class="property-style-tenure"&gt;&lt;/span&gt;\n                &lt;div class="property-details"&gt;\n\n                    \n                                                    &lt;div class="property-price"&gt;\n                                &lt;div class="property-style-tenure"&gt;&lt;span&gt;&lt;/span&gt;&lt;/div&gt;\xa3425&lt;small&gt;/pw&lt;/small&gt;\n                                                                    &lt;span class="fees-link-wrapper"&gt;+ &lt;a target="_blank" href="http://www.ldg.co.uk/residential/property-lettings/fees-and-charges/"&gt;fees&lt;/a&gt;&lt;/span&gt;\n                                                            &lt;/div&gt;\n                        \n                    \n                    &lt;div class="property-features"&gt;\n                                                    &lt;div class="property-feature"&gt;\n                                &lt;div class="property-living_rooms"&gt;\n                                    &lt;span class="esf-icon esf-32 esf-icon-living_rooms"&gt;&lt;/span&gt;\n                                    1                                    Reception                                &lt;/div&gt;\n                            &lt;/div&gt;\n                        \n                                                    &lt;div class="property-feature"&gt;\n                                &lt;div class="property-bedrooms"&gt;\n                                    &lt;span class="esf-icon esf-32 esf-icon-bedrooms"&gt;&lt;/span&gt;\n                                    1                                    Bedroom                                &lt;/div&gt;\n                            &lt;/div&gt;\n                        \n                                                    &lt;div class="property-feature"&gt;\n                                &lt;div class="property-bathrooms"&gt;\n                                    &lt;span class="esf-icon esf-32 esf-icon-bathrooms"&gt;&lt;/span&gt;\n                                    1                                    Bathroom                                &lt;/div&gt;\n                            &lt;/div&gt;\n                                            &lt;/div&gt;&lt;!-- /.property-features --&gt;\n\n\n                        &lt;div class="property-media"&gt;\n                                                              &lt;a href="http://www.ldg.co.uk/wp-content/uploads/2016/09/FLP_3453_1_large-724x1024.png" target="_blank" class="alternative-link fancybox " rel="fancybox-group"&gt;View Floor Plan&lt;/a&gt;\n      \n                                                                                                &lt;span class="separator"&gt;|&lt;/span&gt;\n                                                                &lt;a href="http://media2.jupix.co.uk/v3/clients/1588/properties/3453/MED_3453_6286.pdf" target="_blank" class="alternative-link"&gt;Download Brochure&lt;/a&gt;\n                                                    &lt;/div&gt;&lt;!-- /.property-media --&gt;\n\n                    &lt;div class="property-read-more"&gt;\n                        &lt;a href="http://www.ldg.co.uk/residential/1-bedroom-property-for-rent-riding-house-street-fitzrovia-london-101588003963/" class="btn btn-sm lighter-dark-primary-color"&gt;\n                            View Details\n                        &lt;/a&gt;\n                    &lt;/div&gt;\n                &lt;/div&gt;&lt;!-- /.property-details --&gt;\n            &lt;/div&gt;&lt;!-- /.property-content --&gt;\n        &lt;/section&gt;\n            &lt;section class="property-card new-post"&gt;\n            &lt;div class="property-thumb"&gt;\n                &lt;a class="property-image" href="http://www.ldg.co.uk/residential/1-bedroom-property-for-rent-grays-inn-road-bloomsbury-london-101588004443/" title="Grays Inn Road, Bloomsbury, London"&gt;\n                    &lt;img src="http://www.ldg.co.uk/wp-content/uploads/2016/08/IMG_3933_1_large.jpg" alt="Grays Inn Road, Bloomsbury, London thumbnail"&gt;\n\n                                    &lt;/a&gt;\n            &lt;/div&gt;&lt;!-- /.property-thumb --&gt;\n\n            &lt;div class="property-content"&gt;\n                                    &lt;header class="property-title"&gt;\n                        &lt;h2&gt;\n                            &lt;a  href="http://www.ldg.co.uk/residential/1-bedroom-property-for-rent-grays-inn-road-bloomsbury-london-101588004443/"&gt;Grays Inn Road, Bloomsbury, London&lt;/a&gt;\n                        &lt;/h2&gt;\n                    &lt;/header&gt;\n                \n                &lt;span class="property-style-tenure"&gt;&lt;/span&gt;\n                &lt;div class="property-details"&gt;\n\n                    \n                                                    &lt;div class="property-price"&gt;\n                                &lt;div class="property-style-tenure"&gt;&lt;span&gt;&lt;/span&gt;&lt;/div&gt;\xa3430&lt;small&gt;/pw&lt;/small&gt;\n                                                                    &lt;span class="fees-link-wrapper"&gt;+ &lt;a target="_blank" href="http://www.ldg.co.uk/residential/property-lettings/fees-and-charges/"&gt;fees&lt;/a&gt;&lt;/span&gt;\n                                                            &lt;/div&gt;\n                        \n                    \n                    &lt;div class="property-features"&gt;\n                                                    &lt;div class="property-feature"&gt;\n                                &lt;div class="property-living_rooms"&gt;\n                                    &lt;span class="esf-icon esf-32 esf-icon-living_rooms"&gt;&lt;/span&gt;\n                                    1                                    Reception                                &lt;/div&gt;\n                            &lt;/div&gt;\n                        \n                                                    &lt;div class="property-feature"&gt;\n                                &lt;div class="property-bedrooms"&gt;\n                                    &lt;span class="esf-icon esf-32 esf-icon-bedrooms"&gt;&lt;/span&gt;\n                                    1                                    Bedroom                                &lt;/div&gt;\n                            &lt;/div&gt;\n                        \n                                                    &lt;div class="property-feature"&gt;\n                                &lt;div class="property-bathrooms"&gt;\n                                    &lt;span class="esf-icon esf-32 esf-icon-bathrooms"&gt;&lt;/span&gt;\n                                    1                                    Bathroom                                &lt;/div&gt;\n                            &lt;/div&gt;\n                                            &lt;/div&gt;&lt;!-- /.property-features --&gt;\n\n\n                        &lt;div class="property-media"&gt;\n                                                        \n                                                                                            &lt;a href="http://media2.jupix.co.uk/v3/clients/1588/properties/3933/MED_3933_5539.pdf" target="_blank" class="alternative-link"&gt;Download Brochure&lt;/a&gt;\n                                                    &lt;/div&gt;&lt;!-- /.property-media --&gt;\n\n                    &lt;div class="property-read-more"&gt;\n                        &lt;a href="http://www.ldg.co.uk/residential/1-bedroom-property-for-rent-grays-inn-road-bloomsbury-london-101588004443/" class="btn btn-sm lighter-dark-primary-color"&gt;\n                            View Details\n                        &lt;/a&gt;\n                    &lt;/div&gt;\n                &lt;/div&gt;&lt;!-- /.property-details --&gt;\n            &lt;/div&gt;&lt;!-- /.property-content --&gt;\n        &lt;/section&gt;\n    ', u'wpp_query': {u'starting_row': 10, u'pagination': u'off', u'show_layout_toggle': False, u'current_page': u'2', u'requested_page': u'2', u'show_children': u'true', u'sortable_attrs': {u'menu_order': u'Default'}, u'sort_by': u'price_rent', u'sort_order': u'ASC', u'ajax_call': True, u'template': u'ajax', u'per_page': u'10', u'query': {u'sort_by': u'price_rent', u'pagi': u'10--10', u'listing_type': u'rent', u'sort_order': u'ASC', u'property_category': u'residential'}, u'sorter': u'', u'disable_wrapper': u'true', u'properties': {u'total': 60, u'results': [u'793240', u'836654', u'793035', u'793044', u'793078', u'793307', u'792965', u'793054', u'792811', u'793344']}`}}
</code></pre>

<p>The extra backslashes are there to escape the quotes etc.. Once you <code>json.loads()</code> the content the extra slashes so in your case call loads on the body:</p>

<pre><code> import json

 request = FormRequest(url, formdata = data)
 js = json.loads(fetch(request).body)
</code></pre>

<p>And to just get the html you would use the key <code>html = js["display"]</code>.</p>

python python-2.7 xpath web-scraping scrapy