UTF-16 codepoint counting in python
<p>I'm getting some data from an API (telegram-bot) I'm using.
I'm using the <a href="https://github.com/python-telegram-bot/python-telegram-bot" rel="nofollow">python-telegram-bot</a> library which interacts with the <a href="https://core.telegram.org/bots/api" rel="nofollow">Telegram Bot api</a>.
The data is returned in the UTF-8 encoding in JSON format.
Example (snippet):</p>

<pre><code>{'message': {'text': 'í ½í±¨\u200dí ½í±©\u200dí ½í±¦\u200dí ½í±¦http://google.com/Ã¦Ã¸Ã¥', 'entities': [{'type': 'url', 'length': 21, 'offset': 11}], 'message_id': 2655}}
</code></pre>

<p>It can be seen that 'entities' contains a single entity of type url and it has a length and an offset.
Now say I wanted to extract the url of the link in the 'text' attribute:</p>

<pre><code>data = {'message': {'text': 'í ½í±¨\u200dí ½í±©\u200dí ½í±¦\u200dí ½í±¦http://google.com/Ã¦Ã¸Ã¥', 'entities': [{'type': 'url', 'length': 21, 'offset': 11}], 'message_id': 2655}}
entities = data['entities']
for entity in entities:
    start = entity['offset']
    end = start + entity['length']
    print('Url: ', text[start:end])
</code></pre>

<p>The code above, however, returns: <code>'://google.com/Ã¦Ã¸Ã¥'</code> which is clearly not the actual url.<br>
The reason for this is that the offset and length are in UTF-16 codepoints. So my question is: Is there any way to work with UTF-16 codepoints in python? I don't need more than to be able to count them.  </p>

<p>I've already tried:</p>

<pre><code>text.encode('utf-8').decode('utf-16')
</code></pre>

<p>But that gives the error: <code>UnicodeDecodeError: 'utf-16-le' codec can't decode byte 0xa5 in position 48: truncated data</code></p>

<p>Any help would be greatly appreciated.
I'm using python 3.5, but since it's for a unified library it would be lovely to get it to work in python 2.x too.</p>

<p>Python has already correctly decoded the UTF-8 encoded JSON data to Python (Unicode) strings, so there is no need to handle UTF-8 here.</p>

<p>You'd have to encode to UTF-16, take the length of the encoded data, and divide by two. I'd encode to either <code>utf-16-le</code> or <code>utf-16-be</code> to prevent a BOM from being added:</p>

<pre><code>&gt;&gt;&gt; len(text.encode('utf-16-le')) // 2
32
</code></pre>

<p>To use the entity offsets, you can encode to UTF-16, slice on <em>doubled</em> offsets, then decode again:</p>

<pre><code>text_utf16 = text.encode('utf-16-le')
for entity in entities:
    start = entity['offset']
    end = start + entity['length']
    entity_text = text_utf16[start * 2:end * 2].decode('utf-16-le')
    print('Url: ', entity_text)
</code></pre>

python python-3.x encoding utf-8 utf-16