Unicode issues when writing to CSV file
<p>I need some guidance, please. I'm using the following code: </p>

<pre><code>import requests
import bs4
import csv

results = requests.get('http://grad-schools.usnews.rankingsandreviews.com/best-graduate-schools/top-engineering-schools/eng-rankings?int=a74509')

reqSoup = bs4.BeautifulSoup(results.text, "html.parser")
i = 0
schools = []

for school in reqSoup:
    x = reqSoup.find_all("a", {"class" : "school-name"})
    while i &lt; len(x):
        for name in x:
            y = x[i].get_text()
            i += 1
            schools.append(y)

with open('usnwr_schools.csv', 'wb') as f:
    writer = csv.writer(f)
        for y in schools:
        writer.writerow([y])
</code></pre>

<p>My problem is that the em-dashes are showing up as utf-8 in the resulting CSV file. I've tried several different things to fix it, but nothing seems to work (including <a href="http://stackoverflow.com/questions/33930767/trying-to-remove-character-em-dash-%C3%A2%E2%82%AC-in-python-using-regex">attempting to use regex</a> to get rid of it, as well as trying the <a href="http://stackoverflow.com/questions/27996448/python-encoding-decoding-problems">.translate method that I found in a StackOverflow</a> question from a few years ago). </p>

<p>What am I missing? I'd like the csv results to just include the text, minus the dashes.</p>

<p>I'm using Python 3.5, and am fairly new to Python.</p>

<p>For removing the dashes try <code>y.replace("â","-").replace("â","-")</code> (first one is em-dash to minus, second one is en-dash to minus)</p>

<p>If you only want ASCII-codepoints you can remove everything else with</p>

<pre><code>import string
whitelist=string.printable+string.whitespace
def clean(s):
    return "".join(c for c in s if c in whitelist)
</code></pre>

<p>(this yields mostly-reasonable results for pure-english text only)</p>

<p>Btw try using</p>

<pre><code>open('usnwr_schools.csv', 'w', newline='', encoding='utf-8') # or whatever encoding you like
</code></pre>

<p>because in Python 3 <code>csv.writer</code> takes text files not binary like it did in Python 2 (you opened it in binary mode (<code>"wb"</code>))</p>

python python-3.x unicode