Byte array is a valid UTF8 encoded String in Java but not in Python
<p>When I run the following in Python 2.7.6, I get an exception:</p>

<pre><code>import base64
some_bytes = b"\x80\x02\x03"
print ("base 64 of the bytes:")
print (base64.b64encode(some_bytes))
try:
    print (some_bytes.decode("utf-8"))
except Exception as e:
    print(e)
</code></pre>

<p>The output:</p>

<pre><code>base 64 of the bytes:
gAID
'utf8' codec can't decode byte 0x80 in position 0: invalid start byte
</code></pre>

<p>So in Python 2.7.6 the bytes represented as <code>gAID</code> are not a valid UTF8.</p>

<p>When I try it in Java 8 (HotSpot 1.8.0_74), using this code:</p>

<pre><code>java.util.Base64.Decoder decoder = java.util.Base64.getDecoder();
byte[] bytes = decoder.decode("gAID");
Charset charset = Charset.forName("UTF8");
String s = new String(bytes, charset);
</code></pre>

<p>I don't get any exception.</p>

<p>How so? Why is the same byte array is valid in Java and invalid in Python, using UTF8 decoding?</p>

<blockquote>
  <p>So in Python 2.7.6 the bytes represented as gAID are not a valid UTF8.</p>
</blockquote>

<p>This is wrong as you try to decode the <code>Base64</code> encoded bytes.</p>

<pre><code>import base64
some_bytes = b"\x80\x02\x03"
print ("base 64 of the bytes:")
print (base64.b64encode(some_bytes))
# store the decoded bytes
some_bytes = base64.b64encode(some_bytes)
decoded_bytes = [hex(ord(c)) for c in some_bytes]
print ("decoded bytes: ")
print (decoded_bytes)
try:
    print (some_bytes.decode("utf-8"))
except Exception as e:
    print(e)
</code></pre>

<p>output</p>

<pre><code>gAID
['0x67', '0x41', '0x49', '0x44']
gAID
</code></pre>

<p>In Java you try to create a <code>String</code> from the Base64 encoded bytes using the UTF-8 charset. Which results (as already answered) in the default replacement character <a href="https://en.wikipedia.org/wiki/Specials_(Unicode_block)#Replacement_character" rel="nofollow">ï¿½</a>.</p>

<p>Running following snippet</p>

<pre><code>java.util.Base64.Decoder decoder = java.util.Base64.getDecoder();
byte[] bytes = decoder.decode("gAID");
System.out.println("base 64 of the bytes:");
for (byte b : bytes) {
    System.out.printf("x%02x ", b);
}
System.out.println();
Charset charset = Charset.forName("UTF8");
String s = new String(bytes, charset);
System.out.println(s);
</code></pre>

<p>produce following output</p>

<pre><code>base 64 of the bytes:
x80 x02 x03 
?
</code></pre>

<p>There you can see the same bytes you are using in the Python snippet. Which lead in Python to <code>'utf8' codec can't decode byte 0x80 in position 0: invalid start byte</code> lead there to an <code>?</code> (it stands for the default replacement character on a not-unicode console)</p>

<p>Following snippet used the bytes from <code>gAID</code> to construct a <code>String</code> with the UTF-8 character set.</p>

<pre><code>byte[] bytes = "gAID".getBytes(StandardCharsets.ISO_8859_1);
for (byte b : bytes) {
    System.out.printf("x%02x ", b);
}
System.out.println();
Charset charset = Charset.forName("UTF8");
String s = new String(bytes, charset);
System.out.println(s);
</code></pre>

<p>output</p>

<pre><code>x67 x41 x49 x44 
gAID
</code></pre>

java python utf-8 character-encoding