regexp_tokenize and Arabic text
<p>I'm using <a href="http://www.nltk.org/_modules/nltk/tokenize/regexp.html" rel="nofollow"><code>regexp_tokenize()</code></a> to return tokens from an Arabic text without any punctuation marks:</p>

<pre><code>import re,string,sys
from nltk.tokenize import  regexp_tokenize

def PreProcess_text(Input):
  tokens=regexp_tokenize(Input, r'[ØØ!.Ø]\s*', gaps=True)
  return tokens

H = raw_input('H:')
Cleand= PreProcess_text(H)
print  '\n'.join(Cleand) 
</code></pre>

<p>It worked fine, but the problem is when I try to print the text.</p>

<p>The output for the text <code>Ø§ÙÙØ§ÙØØ³Ø¹Ø¯</code>:</p>

<pre><code>    ?ÙÙ
    ?Ù
    ?
    ?
    ? 
</code></pre>

<p>but if the text is in English, even with an Arabic punctuation marks, it prints the right result.</p>

<p>The output for the text <code>hiØeman</code>:</p>

<pre><code>     hi
     eman
</code></pre>

<p>When you use <code>raw_input</code>, the symbols are coded as bytes.</p>

<p>You need to convert it into a Unicode string with</p>

<pre><code>H.decode('utf8')
</code></pre>

<p>And you may keep your regex:</p>

<pre><code>tokens=regexp_tokenize(Input, r'[ØØ!.Ø]\s*', gaps=True)
</code></pre>

python regex nltk