Python Writing Weird Unicode to CSV
<p>I'm attempting to extract article information using the python <a href="https://github.com/codelucas/newspaper" rel="nofollow">newspaper3k</a> package and then write to a CSV file. While the info is downloaded correctly, I'm having issues with the output to CSV. I don't think I fully understand unicode, despite my efforts to read about it. </p>

<pre><code>from newspaper import Article, Source
import csv

first_article = Article(url="http://www.bloomberg.com/news/articles/2016-09-07/asian-stock-futures-deviate-as-s-p-500-ends-flat-crude-tops-46")

first_article.download()
if first_article.is_downloaded:
    first_article.parse()
    first_article.nlp

article_array = []
collate = {}

collate['title'] = first_article.title
collate['content'] = first_article.text
collate['keywords'] = first_article.keywords
collate['url'] = first_article.url
collate['summary'] = first_article.summary
print(collate['content'])
article_array.append(collate)

keys = article_array[0].keys()
with open('bloombergtest.csv', 'w') as output_file:
    csv_writer = csv.DictWriter(output_file, keys)
    csv_writer.writeheader()
    csv_writer.writerows(article_array)

output_file.close()
</code></pre>

<p>When I print collate['content'], which is first_article.text, the console outputs the article's content just fine. Everything shows up correctly, apostrophes and all. When I write to the CVS, the content cell text has odd characters in it. For example: </p>

<p>Ã¢â¬ÅAt the end of the day, EuropeÃ¢â¬â¢s economy isnÃ¢â¬â¢t in great shape, inflation doesnÃ¢â¬â¢t look exciting and there are a bunch of political risks to reckon with.</p>

<p>So far I have tried:</p>

<pre><code>with open('bloombergtest.csv', 'w', encoding='utf-8') as output_file:
</code></pre>

<p>to no avail. I also tried utf-16 instead of 8, but that just resulted in the cells writing in an odd order. It didn't create the cells correctly in the CSV, although the output looked correct. I've also tried .encode('utf-8') are various variable but nothing has worked. </p>

<p>What's going on? Why would the console print the text correctly, while the CSV file has odd characters? How can I fix this? </p>

<p>Changing <code>with open('bloombergtest.csv', 'w', encoding='utf-8') as output_file:</code> to <code>with open('bloombergtest.csv', 'w', encoding='utf-8-sig') as output_file:</code>, worked, as recommended by Leon and Mark Tolonen. </p>

python csv unicode