python decode partial utf-8 byte array
<p>I'm getting data from channel which is not aware about UTF-8 rules. So sometimes when UTF-8 is using multiple bytes to code one character and I try to convert part of received data into text I'm getting error during conversion. By nature of interface (stream without any end) I'm not able to find out when data are full. Thus I need to handle partial utf-8 decoding. Basically I need to decode what I can and store partial data. Stored partial data will be added as prefix to next data. My question is if there is some neat function in python to allow it? </p>

<p>[EDIT]
Just to ensure you I know about function in <a href="https://docs.python.org/3/library/stdtypes.html#bytes.decode" rel="nofollow">docs.python</a></p>

<pre><code> bytes.decode(encoding="utf-8", errors="ignore")
</code></pre>

<p>but the issue is it would not return me where is the error and so I can not know how much bytes from end I shall keep.</p>

<p>So far I come up with not so nice function:</p>

<pre><code>def decodeBytesUtf8Safe(toDec):
    """
    decodes byte array in utf8 to string. It can handle case when end of byte array is
    not complete thus making utf8 error. in such case text is translated only up to error.
    Rest of byte array (from error to end) is returned as second parameter and can be
    combined with next byte array and decoded next time.
    :param toDec: bytes array to be decoded a(eg bytes("abc","utf8"))
    :return:
     1. decoded string
     2. rest of byte array which could not be encoded due to error
    """
    okLen = len(toDec)
    outStr = ""
    while(okLen&gt;0):
        try:
            outStr = toDec[:okLen].decode("utf-8")
        except UnicodeDecodeError as ex:
            okLen -= 1
        else:
            break
    return outStr,toDec[okLen:]
</code></pre>

<p>you can test it using script:</p>

<pre><code>def test(arr):
    expStr = arr.decode("utf-8")
    errorCnt = 0
    for i in range(len(arr)+1):
        decodedTxt, rest = decodeBytesUtf8Safe(arr[0:i])
        decodedTxt2, rest2 = decodeBytesUtf8Safe(rest+arr[i:])
        recvString = decodedTxt+decodedTxt2
        sys.stdout.write("%02d ; %s (%s - %s )\n"%(i,recvString,decodedTxt, decodedTxt2))
        if(expStr != recvString):
            print("Error when divided at %i"%(i))
            errorCnt += 1
    return errorCnt

testUtf8 = bytes([0x61, 0xc5, 0xbd, 0x6c, 0x75, 0xc5, 0xa5, 0x6f, 0x75, 0xc4, 0x8d, 0x6b, 0xc3, 0xbd, 0x20, 0x6b, 0xc5, 0xaf, 0xc5, 0x88])
err = test(testUtf8)
print("total errors %i"%(err))
</code></pre>

<p>it shall give you the output:</p>

<pre><code>00 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å ( - aÅ½luÅ¥ouÄkÃ½ kÅ¯Å )
01 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å (a - Å½luÅ¥ouÄkÃ½ kÅ¯Å )
02 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å (a - Å½luÅ¥ouÄkÃ½ kÅ¯Å )
03 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å (aÅ½ - luÅ¥ouÄkÃ½ kÅ¯Å )
04 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å (aÅ½l - uÅ¥ouÄkÃ½ kÅ¯Å )
05 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å (aÅ½lu - Å¥ouÄkÃ½ kÅ¯Å )
06 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å (aÅ½lu - Å¥ouÄkÃ½ kÅ¯Å )
07 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å (aÅ½luÅ¥ - ouÄkÃ½ kÅ¯Å )
08 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å (aÅ½luÅ¥o - uÄkÃ½ kÅ¯Å )
09 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å (aÅ½luÅ¥ou - ÄkÃ½ kÅ¯Å )
10 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å (aÅ½luÅ¥ou - ÄkÃ½ kÅ¯Å )
11 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å (aÅ½luÅ¥ouÄ - kÃ½ kÅ¯Å )
12 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å (aÅ½luÅ¥ouÄk - Ã½ kÅ¯Å )
13 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å (aÅ½luÅ¥ouÄk - Ã½ kÅ¯Å )
14 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å (aÅ½luÅ¥ouÄkÃ½ -  kÅ¯Å )
15 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å (aÅ½luÅ¥ouÄkÃ½  - kÅ¯Å )
16 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å (aÅ½luÅ¥ouÄkÃ½ k - Å¯Å )
17 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å (aÅ½luÅ¥ouÄkÃ½ k - Å¯Å )
18 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å (aÅ½luÅ¥ouÄkÃ½ kÅ¯ - Å )
19 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å (aÅ½luÅ¥ouÄkÃ½ kÅ¯ - Å )
20 ; aÅ½luÅ¥ouÄkÃ½ kÅ¯Å (aÅ½luÅ¥ouÄkÃ½ kÅ¯Å -  )
total errors 0
</code></pre>

python utf-8