lua cjson cannot decode specific unicode char?
<p>I'm getting the following error from lua cjson when trying to decode a specific unicode char,</p>

<pre><code>root@9dc8433e6d83:~/torch-rnn# th train.lua -input_h5 data/aud.h5 -input_json data/aud.json -batch_size 50 -seq_length 100 -rnn_size 256 -max_epochs 50
Running with CUDA on GPU 0  
/root/torch/install/bin/luajit: train.lua:77: Expected value but found invalid unicode escape code at character 350873
stack traceback:
    [C]: in function 'read_json'
    train.lua:77: in main chunk
    [C]: in function 'dofile'
    /root/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
    [C]: at 0x00406670
</code></pre>

<p>by following the source, I can see that train.lua read_json is using cjson under the covers.</p>

<p>The unicode escape code in question is \uda85</p>

<p>If I go to <a href="https://www.branah.com/unicode-converter" rel="nofollow">https://www.branah.com/unicode-converter</a>  it tells me the character the escape should decode to.</p>

<p>The unicode escape was generated using python unichr(55941) and written to a file with PYTHONIOENCODING=UTF-8 via redirection of the python script output.</p>

<p>The following demonstrates how the character was generated;</p>

<pre><code>echo "print unichr(55941)" &gt; test.py
python test.py
Traceback (most recent call last):
  File "test.py", line 1, in &lt;module&gt;
    print unichr(55941)
UnicodeEncodeError: 'ascii' codec can't encode character u'\uda85' in position 0: ordinal not in range(128)

# export PYTHONIOENCODING=UTF-8
# python test.py
ï¿½ï¿½ï¿½
# python test.py &gt; tfile
# cat tfile
ï¿½ï¿½ï¿½
# python
Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; f=open("tfile",'r')
&gt;&gt;&gt; s=f.readline()
&gt;&gt;&gt; s
'\xed\xaa\x85\n'
&gt;&gt;&gt; print s
ï¿½ï¿½ï¿½

&gt;&gt;&gt; s.decode('utf-8')
u'\uda85\n'
</code></pre>

<p>What I am trying to do overall is take a set of integers in the range 0-65535 and using python map them to UTF-8 chars and write them out to a file. Then I would like to use torch-rnn that uses LUA to train an RNN on the sequence of chars. I'm getting the error when attempting to run th train.lua on the files generated by the torch-rnn python scripts/preprocess.py</p>

<p>It seems the problem was unicode surrogates, understanding that means I can filter/switch them for different values. In this use case, thats not such a big problem.</p>

python unicode utf-8 lua torch