Unicode-escaped file processing error
<p>â Not quite. The post linked above contains the solution to a problem that showed up in the comments of this post while trying to solve it, and seems unrelated to the original problem.</p>

<p>I have a raw text file containing only the following line, and no newline:</p>

<pre><code>Q853 \u0410\u043D\u0434\u0440\u0435\u0439 \u0410\u0440\u0441\u0435\u043D\u044C\u0435\u0432\u0438\u0447 \u0422\u0430\u0440\u043A\u043E\u0432\u0441\u043A\u0438\u0439
</code></pre>

<p>The characters are escaped as shown above, meaning that the <code>\u05E9</code> is really a backslash, followed by 5 alphanumeric characters (and not an Unicode character). I am trying to decode the file using the following code:</p>

<pre><code>import codecs

with codecs.open("wikidata-terms20.nt", 'r', encoding='unicode_escape') as input:
    with open("wikidata-terms3.nt", "w") as output:
        for line in input:
            output.write(line)
</code></pre>

<p>Using <code>print</code> is not possible here, see in the comments.</p>

<p>Running it gives me the following error:</p>

<pre><code>Traceback (most recent call last):
  File "terms2.py", line 5, in &lt;module&gt;
    for line in input:
  File "C:\Program Files\Python35\lib\codecs.py", line 711, in __next__
    return next(self.reader)
  File "C:\Program Files\Python35\lib\codecs.py", line 642, in __next__
    line = self.readline()
  File "C:\Program Files\Python35\lib\codecs.py", line 555, in readline
    data = self.read(readsize, firstline=True)
  File "C:\Program Files\Python35\lib\codecs.py", line 501, in read
newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'unicodeescape' codec can't decode bytes in position 67-71: truncated \uXXXX escape
</code></pre>

<p>What is going on?</p>

<p>I am running Python 3.5.1 on Windows 8.1, and the code seems to work for most other Unicode characters (this line is the first one to cause the crash).</p>

<p>See edit history for the original question.</p>

<p>It seems that the data read by the decoder is truncated at (after) character#72 (0-based character #71). That obviously is somehow related to the <a href="https://bugs.python.org/issue10344" rel="nofollow">this bug</a>.</p>

<p>The following code produces the same error as in your example:</p>

<pre><code>open("wikidata-terms20.nt", 'r').readline()
open("wikidata-terms20.nt", 'r').readline(72)
</code></pre>

<p>Increasing the readline size above the actual size of the input or setting it to -1 eliminates the error:</p>

<pre><code>open("wikidata-terms20.nt", 'r').readline(1000)
open("wikidata-terms20.nt", 'r').readline(-1)
</code></pre>

<p>Evidently, <code>for line in input:</code> obtains the line to be decoded with <code>readline()</code>, effectively truncating the data-to-be-decoded to 72 characters.</p>

<p>So here are a couple of workarounds:</p>

<p><strong>Workaround 1:</strong></p>

<pre><code>import codecs

with open("wikidata-terms20.nt", 'r') as input:
    with open("wikidata-terms3.nt", "w") as output:
        for line in input:
            output.write(codecs.decode(line, 'unicode_escape'))
</code></pre>

<p><strong>Workaround 2:</strong></p>

<pre><code>import codecs

with codecs.open("wikidata-terms20.nt", 'r', encoding='unicode_escape') as input:
    with open("wikidata-terms3.nt", "w") as output:
        for line in input.readlines():
            output.write(line)
</code></pre>

python python-3.x unicode python-unicode