UTF-16 Code Units In Python Polyglot
<p>I need to extract the number of UTF-16 code units from the start of the string at which a location name starts from a Python sting. I am using Polyglot NER to tag a location in a Python string. For example, "Obama was born in the United States. But I was born in Alabama", would mark "United States" and "Alabama". The Python Polyglot extractor simply returns to the locations tagged, and the how many words from the front they start. How do I figure out the number of UTF-16 code units from the start of the string the word occurs? </p>

<p>Java interface that requires the information <a href="https://github.com/Berico-Technologies/CLAVIN/blob/master/src/main/java/com/bericotech/clavin/extractor/LocationOccurrence.java" rel="nofollow">https://github.com/Berico-Technologies/CLAVIN/blob/master/src/main/java/com/bericotech/clavin/extractor/LocationOccurrence.java</a></p>

<p>Just to clarify some of @Ignacio Vazquez-Abrams' comments.
When processing or analyzing text you don't want to have to worry about how many bytes a given character takes up.  That's why you take the 'encoding' out of the equation by first 'decoding' the encoded text to a separate text/str representation.</p>

<pre><code>&gt;&gt;&gt; encoded_text = 'hello world'.encode('utf16')
&gt;&gt;&gt; encoded_text
b'\xff\xfeh\x00e\x00l\x00l\x00o\x00 \x00w\x00o\x00r\x00l\x00d\x00'
&gt;&gt;&gt; type(encoded_text)
&lt;class 'bytes'&gt;
&gt;&gt;&gt; len(encoded_text)
24


&gt;&gt;&gt; decoded_text = encoded_text.decode('utf16')
&gt;&gt;&gt; decoded_text
'hello world'
&gt;&gt;&gt; type(decoded_text)
&lt;class 'str'&gt;
&gt;&gt;&gt;
&gt;&gt;&gt; len(decoded_text)
11
</code></pre>

<p>I did see the <code>UTF-16 code units</code> in the java code you posted... </p>

<p>You could do something like this to get the number of bytes from the start:</p>

<pre><code>sentence = "Obama was born in the United States. But I was born in Alabama".encode('UTF-16LE')
word = 'United States'.encode('UTF-16LE')

bytes_from_start = None
for start_byte_position in range(len(sentence)):
    candidate = sentence[start_byte_position: start_byte_position + len(word)]
    if word == candidate:
        bytes_from_start = len(sentence[:start_byte_position])
        print('bytes from start: ', bytes_from_start)
        print('len(sentence[:start_byte_position]): ', len(sentence[:start_byte_position]))
        print('Preceding text: "{}"'.format(sentence[:start_byte_position].decode('UTF-16LE')))
        break
</code></pre>

<p>But it's still not clear if <em>UTF-16 code units</em> == <em>bytes</em>.  I have a feeling it really just wants the number of characters from the start.  And if that's all you need you can use the str object's .index() method:</p>

<pre><code>sentence = "Obama was born in the United States. But I was born in Alabama"
word = 'United States'
characters_from_start = sentence.index(word)
</code></pre>

python utf-8 utf polyglot