scikit-learn CountVectorizer UnicodeDecodeError
<p>I have the following code snippet where I'm trying to list the term frequencies, where <code>first_text</code> and <code>second_text</code> are <code>.tex</code> documents:</p>

<pre><code>from sklearn.feature_extraction.text import CountVectorizer
training_documents = (first_text, second_text)  
vectorizer = CountVectorizer()
vectorizer.fit_transform(training_documents)
print "Vocabulary:", vectorizer.vocabulary 
</code></pre>

<p>When I run the script, I get the following:</p>

<pre><code>File "test.py", line 19, in &lt;module&gt;
    vectorizer.fit_transform(training_documents)
  File "/usr/local/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 817, in fit_transform
    self.fixed_vocabulary_)
  File "/usr/local/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 752, in _count_vocab
    for feature in analyze(doc):
  File "/usr/local/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 238, in &lt;lambda&gt;
    tokenize(preprocess(self.decode(doc))), stop_words)
  File "/usr/local/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 115, in decode
    doc = doc.decode(self.encoding, self.decode_error)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa2 in position 200086: invalid start byte
</code></pre>

<p>How can I fix this issue?</p>

<p>Thanks.</p>

<p>If you can work out what the encoding of your documents is (maybe they are <code>latin-1</code>) you can pass this to <code>CountVectorizer</code> with</p>

<pre><code>vectorizer = CountVectorizer(encoding='latin-1')
</code></pre>

<p>Otherwise you can just skip the tokens containing the problematic bytes with</p>

<pre><code>vectorizer = CountVectorizer(decode_error='ignore')
</code></pre>

python scikit-learn