Dictionary keys cannot be encoded as utf-8
<p>I am using the twitter streaming api (tweepy) to capture several tweets. I do this in python2.7. </p>

<p>After I have collected a corpus of tweets I break each tweet into words and add each word to a dictionary as keys, where the values are the participation of each word in <code>positive</code> or <code>negative</code> sentences.</p>

<p>When I retrieve the words as keys of the dictionary and try to process them for a next iteration I get </p>

<blockquote>
  <p>UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2: ordinal not in range(128)</p>
</blockquote>

<p>errors</p>

<p>The weird thing is that before I place them as dictionary keys I encode them without errors. Here is a sample code</p>

<pre><code>pos = {}
neg = {}
for status in corpus:
    p = s.analyze(status).polarity
    words = []
    # gather real words
    for w in status.split(' '):
        try:
            words.append(w.encode('utf-8'))
        except UnicodeDecodeError as e:
            print(e)
    # assign sentiment of the sentence to the words
    for w in words:
        if w not in pos:
            pos[w] = 0
            neg[w] = 0

        if p &gt;= 0:                    
            pos[w] += 1
        else:
            neg[w] += 1

k = pos.keys()
k = [i.encode('utf-8') for i in k]  # &lt;-- for this line a get an error
p = [v for i, v in pos.items()]
n = [v for i, v in neg.items()]
</code></pre>

<p>So this piece of code will catch no errors during the splitting of the words but it will throw an error when trying to encode the keys again. I should note than normally I wouldn't try to encode the keys again, as I would think they are already properly encoded. But I added this extra encoding to narrow down the source of the error. </p>

<p>Am I missing something? Do you see anything wrong with my code?</p>

<p>to avoid confusion here is a sample code more close to the original that is not trying to encode the keys again</p>

<pre><code>k = ['happy']
for i in range(3):
    print('sampling twitter --&gt; {}'.format(i))
    myStream.filter(track=k)  # &lt;-- this is where I will receive the error in the second iteration
    for status in corpus:
        p = s.analyze(status).polarity
        words = []
        # gather real words
        for w in status.split(' '):
            try:
                words.append(w.encode('utf-8'))
            except UnicodeDecodeError as e:
                print(e)
        # assign sentiment of the sentence to the words
        for w in words:
            if w not in pos:
                pos[w] = 0
                neg[w] = 0

            if p &gt;= 0:                    
                pos[w] += 1
            else:
                neg[w] += 1

    k = pos.keys()
</code></pre>

<p>(<strong>please suggest a better title for the question</strong>)</p>

<p>You get a <strong>decode</strong> error while you are trying to <strong>encode</strong> a string. This seems weird but it is due to implicit decode/encode mechanism of Python.</p>

<p>Python allows to encode strings to obtain bytes and decode bytes to obtain strings. This means that Python can encode only strings and decode only bytes.</p>

<p>So when you try to encode bytes, Python (which does not know how to encode bytes) tries to implicitely decode the byte to obtain a string to encode and it uses its default encoding to do that.
This is why you get a decode error while trying to encode something: the implicit decoding.</p>

<p>That means that you are probably trying to encode something which is already encoded.</p>

python python-2.7 encoding utf-8 tweepy