Error while stemming arabic text in file using ISRIStemmer
<p>I am trying to stem the contents of a text file (text.txt) in Arabic Language using nltk.stem.isri .
The text.txt file contains the following Arabic text:</p>

<p>ØªØªÙÙØ¹ Ø§ÙÙØ´Ø§Ø¹Ø± Ø§ÙØªÙ ÙØ´Ø¹Ø± Ø¨ÙØ§ Ø§ÙØ¥ÙØ³Ø§Ù Ø®ÙØ§Ù Ø­ÙØ§ØªÙØ ÙØªÙÙÙ ÙØ°Ù Ø§ÙÙØ´Ø§Ø¹Ø± ØªØ¨Ø¹Ø§Ù ÙÙÙØ§ÙÙ Ø£Ù Ø£Ø´Ø®Ø§Øµ ÙØ¹ÙÙÙÙØ ÙÙØ¹ØªØ¨Ø± Ø§ÙØ­Ø¨ Ø£Ø­Ø¯ Ø§ÙÙØ´Ø§Ø¹Ø± Ø§ÙØ¬ÙÙÙØ© Ø§ÙØªÙ ØªØµÙØ¨ Ø§ÙØ¥ÙØ³Ø§Ù ÙØªØºÙØ± ÙÙ ÙØ¸Ø±ØªÙ ÙÙØ¹Ø§ÙÙ ÙÙ Ø­ÙÙÙ ÙÙØµØ¨Ø­ Ø¥ÙØ¬Ø§Ø¨ÙØ§Ù Ø£ÙØ«Ø±Ø ÙÙÙ Ø§ÙØ¹ÙØ§ÙØ© Ø§ÙØ¹Ø§Ø·ÙÙØ© Ø§ÙØªÙ ØªØ¬ÙØ¹ Ø¨ÙÙ Ø§ÙØ±Ø¬Ù ÙØ§ÙÙØ±Ø£Ø© Ø£Ù Ø¨ÙÙ Ø§ÙØ£Ø´Ø®Ø§Øµ Ø§ÙÙØ®ØªÙÙÙÙ ÙÙ Ø­ÙØ§ØªÙØ ÙÙØ§ ØªØ¬Ø±Ù ÙÙ ÙØ´Ø§Ø¹Ø± Ø§ÙØ§ÙØªÙØ§Ù ÙØ§ÙÙÙØ¯Ø© ÙØ§ÙÙØ·Ù. Ø§ÙØ­Ø¨ Ø£ÙØ¶Ø§Ù Ø­Ø§ÙØ©Ù ÙÙ Ø§ÙØ§ÙØ¬Ø°Ø§Ø¨ ÙØ§ÙØ¥Ø¹Ø¬Ø§Ø¨ Ø¨ÙÙ Ø§ÙØ£Ø´Ø®Ø§ØµØ ÙÙÙØ§Ù Ø¨Ø£ÙÙ ÙÙØ¹Ù ÙÙ Ø§ÙÙÙÙÙØ§Ø¡ Ø§ÙÙØªØ¨Ø§Ø¯ÙØ©Ø Ø¥Ø° ÙØ¹Ø¨Ø± Ø¹Ù ÙÙØ¹Ù ÙÙ Ø§ÙØªÙØ§Ø¹Ù Ø¨ÙÙ Ø§ÙÙØ§Ø³Ø ÙÙÙ Ø§ÙØ­Ø¨ ÙÙØ±Ø² Ø§ÙØ¬Ø³Ù ÙØ±ÙÙÙ Ø§ÙØ£ÙÙØ³ÙØªÙØ³ÙÙ ÙÙÙ Ø§ÙÙØ±ÙÙÙ Ø§ÙØ°Ù ÙØ¯Ø¹Ù Ø¨ÙØ±ÙÙÙ Ø§ÙÙØ­Ø¨ÙÙ ÙØ§ÙÙØ­Ø¨Ø© Ø§ÙØ°Ù ÙÙØ±Ø²Ù Ø§ÙØ¬Ø³Ù ÙÙØ± Ø§ÙÙÙØ§Ø¡ Ø¨ÙÙ Ø§ÙØ£Ø­Ø¨Ø©Ø ÙÙÙÙØ§ ÙÙÙ ÙÙ Ø³Ø·ÙØ±Ù Ø³ÙØªØ­Ø¯Ø« Ø¹Ù Ø¨Ø¹Ø¶ Ø§ÙÙØ¹ÙÙÙØ§Øª Ø§ÙØ¹Ø§ÙØ© Ø­ÙÙ Ø§ÙØ­Ø¨ Ø¨Ø´ÙØ¡Ù ÙÙ Ø§ÙØªÙØµÙÙ ÙØ§ÙØªÙØ¶ÙØ­.</p>

<p>I used the following code by referring to a previous question: <a href="http://stackoverflow.com/questions/16835372/python-stemming-words-in-a-file">Python Stemming words in a File</a></p>

<pre><code># -*- coding: UTF-8 -*-

from nltk.stem.isri import ISRIStemmer
def stemming_text_1():
    with open('test.txt', 'r') as f:
        for line in f:
            print line
            singles = []

            stemmer = ISRIStemmer()
            for plural in line.split():
                singles.append(stemmer.stem(plural))
            print ' '.join(singles)

stemming_text_1()
</code></pre>

<p>It prints the contents of the file and this error:</p>

<pre><code>/home/waheeb/anaconda2/lib/python2.7/site-packages/nltk/stem/isri.py:154:     UnicodeWarning: Unicode equal comparison failed to convert     both arguments to Unicode - interpreting them as being unequal
  if token in self.stop_words:
Traceback (most recent call last):
  File "Arabic_stem.py", line 15, in &lt;module&gt;
    stemming_text_1()
  File "Arabic_stem.py", line 12, in stemming_text_1
    singles.append(stemmer.stem(plural))
  File "/home/waheeb/anaconda2/lib/python2.7/site-packages/nltk/stem    /isri.py", line 156, in stem
    token = self.pre32(token)     # remove length three and length two     prefixes in this order
  File "/home/waheeb/anaconda2/lib/python2.7/site-packages/nltk/stem    /isri.py", line 198, in pre32
    if word.startswith(pre3):
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd8 in position 0:     ordinal not in range(128)
</code></pre>

<p>Try decoding the lines from the file to <em>unicode</em> before passing it to the stemmer. I am assuming that your input file is encoded as UTF8 (seems likely looking at the error), however, you can change the encoding as suits:</p>

<pre><code>for line in f:
    line = line.decode('utf8')    # use the correct encoding here
    ...
</code></pre>

<p>Alternatively you can use <a href="https://docs.python.org/2/library/io.html#io.open" rel="nofollow"><code>io.open()</code></a>, specify the encoding, and Python will decode the incoming stream to unicode:</p>

<pre><code>with io.open('test.txt', encoding='utf8') as f:
    ...
</code></pre>

python nltk arabic