how to decode a string containing Persian/Arabic characters?
<p>In web scraping sometimes I need to get data from Persian webpages, so when I try to decode it and see the extracted data, the result is not what I expect to be.</p>

<p>Here is the step-by-step code for when this problem occurs :</p>

<p><strong>1.getting data from a Persian website</strong></p>

<pre><code>import urllib2

data = urllib2.urlopen('http://cafebazar.ir').read() # this is a persian website
</code></pre>

<p><strong>2.detecting type of encoding</strong></p>

<pre><code>import chardet
chardet.detect(data)
# in this case result is : 
{'confidence': 0.6567038227597763, 'encoding': 'ISO-8859-2'}
</code></pre>

<p><strong>3. decoding and encoding</strong></p>

<pre><code>final = data.decode(chardet.detect(data)['encoding']).encode('ascii', 'ignore')
</code></pre>

<p>but the final result is not in Persian at all !</p>

<p>Instead of encoding into ascii, you should decode into something else, for example <code>utf-8</code>:</p>

<pre><code>final = data.decode(chardet.detect(data)['encoding']).encode('utf-8')
</code></pre>

<p>In order to view it though, you should write it into a file as most terminals do not display non-ascii chars correctly:</p>

<pre><code>with open("temp_file.txt", "w", encoding="utf-8") as myfile:
    myfile.write(data.decode(chardet.detect(data)['encoding']))
</code></pre>

python python-2.7 decode python-unicode