UnicodeDecodeError while reading a custom created corpus in NLTK
<p>I made custom corpus for detecting polarity of sentences using nltk module. Here is the hierarchy of the corpus:</p>

<p>polar<br>
--polar<br>
----polar_tweets.txt<br>
--nonpolar<br>
----nonpolar_tweets.txt<br></p>

<p>And here is how I am importing that corpus in my source code:</p>

<pre><code>polarity = LazyCorpusLoader('polar', CategorizedPlaintextCorpusReader, r'(?!\.).*\.txt', cat_pattern=r'(polar|nonpolar)/.*', encoding='utf-8')
corpus = polarity
print(corpus.words(fileids=['nonpolar/non-polar.txt']))
</code></pre>

<p>but it raises the following error:</p>

<pre><code>Traceback (most recent call last):
  File "E:/Analytics Practice/Social Media Analytics/analyticsPlatform/DataAnalysis/SentimentAnalysis/data/training_testing_data.py", line 9, in &lt;module&gt;
    print(corpus.words(fileids=['nonpolar/nonpolar_tweets.txt']))
  File "E:\Analytics Practice\Social Media Analytics\analyticsPlatform\lib\site-packages\nltk\util.py", line 765, in __repr__
    for elt in self:
  File "E:\Analytics Practice\Social Media Analytics\analyticsPlatform\lib\site-packages\nltk\corpus\reader\util.py", line 291, in iterate_from
    tokens = self.read_block(self._stream)
  File "E:\Analytics Practice\Social Media Analytics\analyticsPlatform\lib\site-packages\nltk\corpus\reader\plaintext.py", line 122, in _read_word_block
    words.extend(self._word_tokenizer.tokenize(stream.readline()))
  File "E:\Analytics Practice\Social Media Analytics\analyticsPlatform\lib\site-packages\nltk\data.py", line 1135, in readline
    new_chars = self._read(readsize)
  File "E:\Analytics Practice\Social Media Analytics\analyticsPlatform\lib\site-packages\nltk\data.py", line 1367, in _read
    chars, bytes_decoded = self._incr_decode(bytes)
  File "E:\Analytics Practice\Social Media Analytics\analyticsPlatform\lib\site-packages\nltk\data.py", line 1398, in _incr_decode
    return self.decode(bytes, 'strict')
  File "C:\Users\prabhjot.rai\AppData\Local\Continuum\Anaconda3\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 269: invalid continuation byte
</code></pre>

<p>While creating the files <code>polar_tweets.txt</code> and <code>nonpolar_tweets.txt</code>, I am decoding the files <code>uncleaned_polar_tweets.txt</code> to <code>utf-8</code> and then writing it to the file <code>polar_tweets.txt</code>. Here is the code for that:</p>

<pre><code>with open(path_to_file, "rb") as file:
    output_corpus = clean_text(file.read().decode('utf-8'))['cleaned_corpus']

output_file = open(output_path, "w")
output_file.write(output_corpus)
output_file.close()
</code></pre>

<p>Where output_file is the <code>polar_tweets.txt</code> and <code>nonpolar_tweets.txt</code>.
Where is the error? Since I am encoding in <code>utf-8</code> initially, and then reading also in <code>utf-8</code> by the line</p>

<pre><code>polarity = LazyCorpusLoader('polar', CategorizedPlaintextCorpusReader, r'(?!\.).*\.txt', cat_pattern=r'(polar|nonpolar)/.*', encoding='utf-8')
</code></pre>

<p>If I replace the <code>encoding='utf-8'</code> by <code>encoding='latin-1'</code>, the code works perfect. Where is the issue? Do I need to decode in <code>utf-8</code> also while creating the corpus?</p>

<p>You need to understand that in Python's model, <code>unicode</code> is a kind of data but <code>utf-8</code> is an <em>encoding.</em> They're not the same thing at all. You're reading your raw text, which is apparently in <code>utf-8</code>; cleaning it, then writing it out to your new corpus <em>without specifying an encoding.</em> So you're writing it out in... who knows what encoding. Don't find out, just clean and generate your corpus again specifying the <code>utf-8</code> encoding. </p>

<p>I hope you're doing all this in Python 3; if not, stop right here and switch to Python 3. Then write out the corpus like this:</p>

<pre><code>output_file = open(output_path, "w", encoding="utf-8")
output_file.write(output_corpus)
output_file.close()
</code></pre>

python character-encoding nltk