Python: Correct Way to refer to index of unicode string
<p>Not sure if this is exactly the problem, but I'm trying to insert a  tag on the first letter of a unicode string and it seems that this is not working. Could these be because unicode indices work differently than those of regular strings?</p>

<p>Right now my code is this:</p>

<pre><code>for index, paragraph in enumerate(intro[2:-2]):
    intro[index] = bold_letters(paragraph, 1)

def bold_letters(string, index):
    return "&lt;b&gt;"+string[0]+"&lt;/b&gt;"+string[index:]
</code></pre>

<p>And I'm getting output like this:</p>

<pre><code>&lt;b&gt;?&lt;/b&gt;?×¨× ×××× ×××©×ª×× ××××¨× ×××××ª× ××¨×¦×× × ×× ×¦×× ××©××× ××¨×¥ ××××× ××× ××××× ××. 
</code></pre>

<p>It seems the unicode gets messed up when I try to insert the HTML tag. I tried messing with the insert position but didn't make any progress.</p>

<p>Example desired output (hebrew goes right to left):</p>

<pre><code>&gt;&gt;&gt;first_letter_bold("××§×××")
"××§××&lt;\b&gt;×&lt;b&gt;"
</code></pre>

<p>BTW, this is for Python 2</p>

<p>You are right, indices work over each <code>byte</code> when you are dealing with <em>raw bytes</em> i.e <code>String</code> in <em>Python(2.x)</em>.</p>

<p>To work seamlessly with Unicode data, you need to first let <em>Python(2.x)</em> know that you are dealing with <code>Unicode</code>, then do the string manipulation. You can finally convert it back to raw bytes to keep the behavior abstracted i.e you get <code>String</code> and you return <code>String</code>.</p>

<p>Ideally you should convert all the data from <code>UTF8</code> raw encoding to <code>Unicode</code> object (I am assuming your source encoding is <code>Unicode UTF8</code> because that is the standard used by most applications these days) at the very beginning of your code and convert back to raw bytes at the fag end of code like saving to DB, responding to client etc. Some frameworks might handle that for you so that you don't have to worry. </p>

<pre><code>def bold_letters(string, index):
    string = string.decode('utf8')
    string "&lt;b&gt;"+string[0]+"&lt;/b&gt;"+string[index:]
    return string.encode('utf8')
</code></pre>

<p>This will also work for <em>ASCII</em> because <em>UTF8</em> is a super-set of <em>ASCII</em>. You can understand how Unicode works and in Python specifically better by reading <a href="http://nedbatchelder.com/text/unipain.html" rel="nofollow">http://nedbatchelder.com/text/unipain.html</a></p>

<p><em>Python 3.x</em> <code>String</code> is a <code>Unicode</code> object so you don't have to explicitly do anything.</p>

python unicode python-2.x