How can I avoid JSON percent-encoding and \u-escaping?
<p>When I parse the file</p>

<pre><code>&lt;html&gt;
    &lt;head&gt;&lt;meta charset="UTF-8"&gt;&lt;/head&gt;
    &lt;body&gt;&lt;a href="DÃ¼sseldorf.html"&gt;DÃ¼sseldorf&lt;/a&gt;&lt;/body&gt;
&lt;/html&gt;
</code></pre>

<p>using</p>

<pre><code>item = SimpleItem()
item['name'] = response.xpath('//a/text()')[0].extract()
item["url"] = response.xpath('//a/@href')[0].extract()
return item
</code></pre>

<p>I end up with either <code>\u</code> escapes</p>

<pre><code>[{
    "name": "D\u00fcsseldorf",
    "url": "D\u00fcsseldorf.html"
}]
</code></pre>

<p>or with percent-encoded strings</p>

<pre><code>D%C3%BCsseldorf
</code></pre>

<p>The <a href="http://doc.scrapy.org/en/latest/topics/exporters.html" rel="nofollow">item exporter</a> described <a href="https://groups.google.com/forum/#!topic/scrapy-users/rJcfSFVZ3O4" rel="nofollow">here</a></p>

<pre><code># -*- coding: utf-8 -*-
import json
from scrapy.contrib.exporter import BaseItemExporter

class UnicodeJsonLinesItemExporter(BaseItemExporter):

    def __init__(self, file, **kwargs):
        self._configure(kwargs)
        self.file = file
        self.encoder = json.JSONEncoder(ensure_ascii=False, **kwargs)

    def export_item(self, item):
        itemdict = dict(self._get_serialized_fields(item))
        self.file.write(self.encoder.encode(itemdict) + '\n')
</code></pre>

<p>along with the appropriate feed exporter setting</p>

<pre><code>FEED_EXPORTERS = {
    'json': 'myproj.exporter.UnicodeJsonLinesItemExporter',
}
</code></pre>

<p>do not help.</p>

<p>How do I get a utf-8-encoded JSON output?</p>

<p>I'm reiterating/expanding an <a href="http://stackoverflow.com/q/24884011/704972">unanswered question</a>.</p>

<p><strong>Update</strong>:</p>

<p>Orthogonal to Scrapy, note that without setting</p>

<pre><code>export PYTHONIOENCODING="utf_8"
</code></pre>

<p>running</p>

<pre><code>&gt; echo { \"name\": \"DÃ¼sseldorf\", \"url\": \"DÃ¼sseldorf.html\" } &gt; dorf.json
&gt; python -c'import fileinput, json;print json.dumps(json.loads("".join(fileinput.input())),sort_keys=True, indent=4, ensure_ascii=False)' dorf.json &gt; dorf_pp.json
</code></pre>

<p>will fail with</p>

<pre><code>Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 16: ordinal not in range(128)
</code></pre>

<p><strong>Update</strong></p>

<p>As posted my question was unanswerable. The UnicodeJsonLinesItemExporter works, but another part of the pipeline was the culprit: As a post-process to pretty-print the JSON output, I was was using <code>python -m json.tool in.json &gt; out.json</code>.</p>

<p>this seems to work for me</p>

<pre><code># -*- coding: utf-8 -*-
import scrapy
import urllib

class SimpleItem(scrapy.Item):
    name = scrapy.Field()
    url = scrapy.Field()

class CitiesSpider(scrapy.Spider):
    name = "cities"
    allowed_domains = ["sitercity.info"]
    start_urls = (
        'http://en.sistercity.info/countries/de.html',
    )

    def parse(self, response):
        for a in response.css('a'):
            item = SimpleItem()
            item['name'] = a.css('::text').extract_first()
            item['url'] = urllib.unquote(
                a.css('::attr(href)').extract_first().encode('ascii')
                ).decode('utf8')
            yield item
</code></pre>

<p>using the feed exporter cited in your question, it worked also using another storage</p>

<pre><code># -*- coding: utf-8 -*-
import json
import io
import os
from scrapy.contrib.exporter import BaseItemExporter
from w3lib.url import file_uri_to_path

class CustomFileFeedStorage(object):

    def __init__(self, uri):
        self.path = file_uri_to_path(uri)

    def open(self, spider):
        dirname = os.path.dirname(self.path)
        if dirname and not os.path.exists(dirname):
            os.makedirs(dirname)
        return io.open(self.path, mode='ab')

    def store(self, file):
        file.close()

class UnicodeJsonLinesItemExporter(BaseItemExporter):

    def __init__(self, file, **kwargs):
        self._configure(kwargs)
        self.file = file
        self.encoder = json.JSONEncoder(ensure_ascii=False, **kwargs)

    def export_item(self, item):
        itemdict = dict(self._get_serialized_fields(item))
        self.file.write(self.encoder.encode(itemdict) + '\n')
</code></pre>

<p>(removing the comments if necessary)</p>

<pre><code>FEED_EXPORTERS = {
    'json': 'myproj.exporter.UnicodeJsonLinesItemExporter'
}
#FEED_STORAGES = {
#   '': 'myproj.exporter.CustomFileFeedStorage'
#}
FEED_FORMAT = 'json'
FEED_URI = "out.json"
</code></pre>

python json unicode scrapy