Python - Reading a UTF-8 encoded string byte-by-byte
<p>I have a device that returns a UTF-8 encoded string. I can only read from it byte-by-byte and the read is terminated by a byte of value 0x00.</p>

<p>I'm making a Python 2.7 function for others to access my device and return string.</p>

<p>In a previous design when the device just returned ASCII, I used this in a loop:</p>

<pre><code>x = read_next_byte()
if x == 0:
    break
my_string += chr(x)
</code></pre>

<p>Where x is the latest byte value read from the device.</p>

<p>Now the device can return a UTF-8 encoded string, but I'm not sure how to convert the bytes that I get back into a UTF-8 encoded string/unicode. </p>

<p><code>chr(x)</code> understandably causes an error when the x>127, so I thought that using <code>unichr(x)</code> may work, but that assumes the value passed is a full unicode character value, but I only have a part 0-255.</p>

<p>So how can I convert the bytes that I get back from the device into a string that can be used in Python and still handle the full UTF-8 string?</p>

<p>Likewise, if I was given a UTF-8 string in Python, how would I break that down into individual bytes to send to my device and still maintain UTF-8?</p>

<p>The correct solution would be to read until you hit the terminating byte, then convert to UTF-8 at that time (so you have all characters):</p>

<pre><code>mybytes = bytearray()
while True:
    x = read_next_byte()
    if x == 0:
        break
    mybytes.append(x)
my_string = mybytes.decode('utf-8')
</code></pre>

<p>The above is the most direct translation of your original code. Interestingly, this is one of those cases where <a href="https://docs.python.org/3/library/functions.html#iter" rel="nofollow">two arg <code>iter</code></a> can be used to dramatically simplify the code by making your C-style stateful byte reader function into a Python iterator that lets you one-line the work:</p>

<pre><code># If this were Python 3 code, you'd use the bytes constructor instead of bytearray
my_string = bytearray(iter(read_next_byte, 0)).decode('utf-8')
</code></pre>

python python-2.7 unicode encoding utf-8