python decode partial utf-8 byte array
<p>I'm getting data from channel which is not aware about UTF-8 rules. So sometimes when UTF-8 is using multiple bytes to code one character and I try to convert part of received data into text I'm getting error during conversion. By nature of interface (stream without any end) I'm not able to find out when data are full. Thus I need to handle partial utf-8 decoding. Basically I need to decode what I can and store partial data. Stored partial data will be added as prefix to next data. My question is if there is some neat function in python to allow it? </p>

<p>[EDIT]
Just to ensure you I know about function in <a href="https://docs.python.org/3/library/stdtypes.html#bytes.decode" rel="nofollow">docs.python</a></p>

<pre><code> bytes.decode(encoding="utf-8", errors="ignore")
</code></pre>

<p>but the issue is it would not return me where is the error and so I can not know how much bytes from end I shall keep.</p>

<p>You can call the codecs module to the rescue. It gives you directly a incremental decoder, that does exactly what you need:</p>

<pre><code>import codecs

dec = codecs.getincrementaldecoder('utf8')()
</code></pre>

<p>You can feed it with: <code>dec.decode(input)</code> and when it is over, optionally add a <code>dec.decode(bytes(), True)</code> to force it to cleanup any stored state.</p>

<p>The test becomes:</p>

<pre><code>&gt;&gt;&gt; def test(arr):
    dec = codecs.getincrementaldecoder('utf8')()
    recvString = ""
    for i in range(len(arr)):
        recvString += dec.decode(arr[i:i+1])
        sys.stdout.write("%02d : %s\n" % (i, recvString))
    recvString += dec.decode(bytes(), True) # will choke on incomplete input...
    return recvString == arr.decode('utf8')

&gt;&gt;&gt; testUtf8 = bytes([0x61, 0xc5, 0xbd, 0x6c, 0x75, 0xc5, 0xa5, 0x6f, 0x75, 0xc4, 0x8d, 0x6b, 0xc3, 0xbd, 0x20, 0x6b, 0xc5, 0xaf, 0xc5, 0x88])
&gt;&gt;&gt; test(testUtf8)
00 : a
01 : a
02 : aÅ½
03 : aÅ½l
04 : aÅ½lu
05 : aÅ½lu
06 : aÅ½luÅ¥
07 : aÅ½luÅ¥o
08 : aÅ½luÅ¥ou
09 : aÅ½luÅ¥ou
10 : aÅ½luÅ¥ouÄ
11 : aÅ½luÅ¥ouÄk
12 : aÅ½luÅ¥ouÄk
13 : aÅ½luÅ¥ouÄkÃ½
14 : aÅ½luÅ¥ouÄkÃ½ 
15 : aÅ½luÅ¥ouÄkÃ½ k
16 : aÅ½luÅ¥ouÄkÃ½ k
17 : aÅ½luÅ¥ouÄkÃ½ kÅ¯
18 : aÅ½luÅ¥ouÄkÃ½ kÅ¯
19 : aÅ½luÅ¥ouÄkÃ½ kÅ¯Å
True
</code></pre>

python utf-8