why output list is empty in my code in Python 2.7
<p>Using Python 2.7 and trying to do simple tokenization on UTF-8 encoded files. The output of <code>a</code> seems a byte string, which is expected, since after <code>tk[0].encode('utf-8')</code>, it converts from Python <code>unicode</code> type to <code>str/byte</code>. My major confusion is why output of <code>b</code> is empty list? I think without encoding (I mean without calling <code>.encode('utf-8')</code>), it should be raw unicode character (e.g. I expect some Chinese character printed, as <code>1.txt</code> is UTF-8 encoded Chinese character file).</p>

<p><strong>Source code</strong>,</p>

<pre><code>import jieba

if __name__ == "__main__":

    with open('1.txt', 'r') as content_file:
        content = content_file.read()

    segment_list = jieba.tokenize(content.decode('utf-8'), mode='search')
    if segment_list is None:
        print 'segment is None'
    else:
        a = [tk[0].encode('utf-8') for tk in segment_list]
        b = [tk[0] for tk in segment_list]

        print a
        print b
</code></pre>

<p>** Output **,</p>

<pre><code>['\xe4\xb8\x8a\xe6\xb5\xb7', '\xe6\xb5\xb7\xe5\xb8\x82', '\xe4\xb8\x8a\xe6\xb5\xb7\xe5\xb8\x82', '\xe6\xb7\xb1\xe5\x9c\xb3', '\xe6\xb7\xb1\xe5\x9c\xb3\xe5\xb8\x82', '\xe7\xa6\x8f\xe7\x94\xb0', '\xe7\xa6\x8f\xe7\x94\xb0\xe5\x8c\xba', '\xe6\xa2\x85\xe6\x9e\x97', '\xe6\x9e\x97\xe8\xb7\xaf', '\xe6\xa2\x85\xe6\x9e\x97\xe8\xb7\xaf', '\xe4\xb8\x8a\xe6\xb5\xb7', '\xe6\xb5\xb7\xe5\xb8\x82', '\xe6\xb5\xa6\xe4\xb8\x9c', '\xe6\x96\xb0\xe5\x8c\xba', '\xe4\xb8\x8a\xe6\xb5\xb7\xe5\xb8\x82', '\xe4\xb8\x8a\xe6\xb5\xb7\xe5\xb8\x82\xe6\xb5\xa6\xe4\xb8\x9c\xe6\x96\xb0\xe5\x8c\xba', '\xe8\x80\x80\xe5\x8d\x8e', '\xe8\xb7\xaf', '\r\n']
[]
</code></pre>

<p>It appears that <code>jieba.tokenize()</code> returns a generator. A generator can be iterated over only once. Better do</p>

<pre><code>    b = [tk[0] for tk in segment_list]
    a = [tk.encode('utf-8') for tk in b]
</code></pre>

python python-2.7 unicode