String encode/decode issue - missing character from end
<p>I am having <code>NVARCHAR</code> type column in my database. I am unable to convert the content of this column to plain string in my code. (I am using <code>pyodbc</code> for the database connection).</p>

<pre><code># This unicode string is returned by the database
&gt;&gt;&gt; my_string = u'\u4157\u4347\u6e65\u6574\u2d72\u3430\u3931\u3530\u3731\u3539\u3533\u3631\u3630\u3530\u3330\u322d\u3130\u3036\u3036\u3135\u3432\u3538\u2d37\u3134\u3039\u352d'

# prints something in chineese 
&gt;&gt;&gt; print my_string
ääæ¹¥æ´âµ²ã°ã¤±ã°ã±ã¹ã³ã±ã°ã°ã°ã­ã°ã¶ã¶ãµã²ã¸â´·ã´ã¹ã­
</code></pre>

<p>The closest I have gone is via encoding it to <code>utf-16</code> as:</p>

<pre><code>&gt;&gt;&gt; my_string.encode('utf-16')
'\xff\xfeWAGCenter-04190517953516060503-20160605124857-4190-5'
&gt;&gt;&gt; print my_string.encode('utf-16')
ï¿½ï¿½WAGCenter-04190517953516060503-20160605124857-4190-5
</code></pre>

<p>But the actual value that I need as per the value store in database is:</p>

<pre><code>WAGCenter-04190517953516060503-20160605124857-4190-51
</code></pre>

<p>I tried with encoding it to <code>utf-8</code>, <code>utf-16</code>, <code>ascii</code>, <code>utf-32</code> but nothing seemed to work.</p>

<p>Does anyone have the idea regarding what I am missing? And how to get the desired result from the <code>my_string</code>.</p>

<p><strong>Edit</strong>: <em>On converting it to <code>utf-16-le</code>, I am able to remove unwanted characters from start, but still one character is missing from end</em></p>

<pre><code>&gt;&gt;&gt; print t.encode('utf-16-le')
WAGCenter-04190517953516060503-20160605124857-4190-5
</code></pre>

<p><em>On trying for some other columns, it is working. <strong>What might be the cause of this intermittent issue?</em></strong></p>

<p>You have a major problem in your database definition, in the way you store values in it, or in the way you read values from it. I can only explain what you are seeing, but neither why nor how to fix it without:</p>

<ul>
<li>the type of the database</li>
<li>the way you input values in it</li>
<li>the way you extract values to obtain your <em>pseudo unicode</em> string</li>
<li>the actual content if you use direct (<em>native</em>) database access</li>
</ul>

<p>What you get is an ASCII string, where the 8 bits characters are grouped by pair to build 16 bit unicode characters in little endian order. As the expected string has an odd numbers of characters, the last character was (irremediably) lost in translation, because the original string ends with <code>u'\352d'</code> where 0x2d is ASCII code for <code>'-'</code> and 0x35 for <code>'5'</code>. Demo:</p>

<pre><code>def cvt(ustring):
    l = []
    for uc in ustring:
        l.append(chr(ord(uc) &amp; 0xFF)) # low order byte
        l.append(chr((ord(uc) &gt;&gt; 8) &amp; 0xFF)) # high order byte
    return ''.join(l)

cvt(my_string)
'WAGCenter-04190517953516060503-20160605124857-4190-5'
</code></pre>

python python-2.7 encode pyodbc nvarchar