Dictionary keys cannot be encoded as utf-8
<p>I am using the twitter streaming api (tweepy) to capture several tweets. I do this in python2.7. </p>

<p>After I have collected a corpus of tweets I break each tweet into words and add each word to a dictionary as keys, where the values are the participation of each word in <code>positive</code> or <code>negative</code> sentences.</p>

<p>When I retrieve the words as keys of the dictionary and try to process them for a next iteration I get </p>

<blockquote>
  <p>UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2: ordinal not in range(128)</p>
</blockquote>

<p>errors</p>

<p>The weird thing is that before I place them as dictionary keys I encode them without errors. Here is a sample code</p>

<pre><code>pos = {}
neg = {}
for status in corpus:
    p = s.analyze(status).polarity
    words = []
    # gather real words
    for w in status.split(' '):
        try:
            words.append(w.encode('utf-8'))
        except UnicodeDecodeError as e:
            print(e)
    # assign sentiment of the sentence to the words
    for w in words:
        if w not in pos:
            pos[w] = 0
            neg[w] = 0

        if p &gt;= 0:                    
            pos[w] += 1
        else:
            neg[w] += 1

k = pos.keys()
k = [i.encode('utf-8') for i in k]  # &lt;-- for this line a get an error
p = [v for i, v in pos.items()]
n = [v for i, v in neg.items()]
</code></pre>

<p>So this piece of code will catch no errors during the splitting of the words but it will throw an error when trying to encode the keys again. I should note than normally I wouldn't try to encode the keys again, as I would think they are already properly encoded. But I added this extra encoding to narrow down the source of the error. </p>

<p>Am I missing something? Do you see anything wrong with my code?</p>

<p>to avoid confusion here is a sample code more close to the original that is not trying to encode the keys again</p>

<pre><code>k = ['happy']
for i in range(3):
    print('sampling twitter --&gt; {}'.format(i))
    myStream.filter(track=k)  # &lt;-- this is where I will receive the error in the second iteration
    for status in corpus:
        p = s.analyze(status).polarity
        words = []
        # gather real words
        for w in status.split(' '):
            try:
                words.append(w.encode('utf-8'))
            except UnicodeDecodeError as e:
                print(e)
        # assign sentiment of the sentence to the words
        for w in words:
            if w not in pos:
                pos[w] = 0
                neg[w] = 0

            if p &gt;= 0:                    
                pos[w] += 1
            else:
                neg[w] += 1

    k = pos.keys()
</code></pre>

<p>(<strong>please suggest a better title for the question</strong>)</p>

<p>Note that the error message says "'ascii' codec can't <strong>decode</strong> ...". That's because when you call <code>encode</code> on something that is already a bytestring in Python 2, it tries to decode it to Unicode first using the default codec.</p>

<p>I'm not sure why you thought that encoding again would be a good idea. Don't do it; the strings are already byetestrings, leave them as that.</p>

python python-2.7 encoding utf-8 tweepy