Unicode issues when writing to CSV file
<p>I need some guidance, please. I'm using the following code: </p>

<pre><code>import requests
import bs4
import csv

results = requests.get('http://grad-schools.usnews.rankingsandreviews.com/best-graduate-schools/top-engineering-schools/eng-rankings?int=a74509')

reqSoup = bs4.BeautifulSoup(results.text, "html.parser")
i = 0
schools = []

for school in reqSoup:
    x = reqSoup.find_all("a", {"class" : "school-name"})
    while i &lt; len(x):
        for name in x:
            y = x[i].get_text()
            i += 1
            schools.append(y)

with open('usnwr_schools.csv', 'wb') as f:
    writer = csv.writer(f)
        for y in schools:
        writer.writerow([y])
</code></pre>

<p>My problem is that the em-dashes are showing up as utf-8 in the resulting CSV file. I've tried several different things to fix it, but nothing seems to work (including <a href="http://stackoverflow.com/questions/33930767/trying-to-remove-character-em-dash-%C3%A2%E2%82%AC-in-python-using-regex">attempting to use regex</a> to get rid of it, as well as trying the <a href="http://stackoverflow.com/questions/27996448/python-encoding-decoding-problems">.translate method that I found in a StackOverflow</a> question from a few years ago). </p>

<p>What am I missing? I'd like the csv results to just include the text, minus the dashes.</p>

<p>I'm using Python 3.5, and am fairly new to Python.</p>

<p>Learn to embrace Unicode...the world isn't ASCII anymore.</p>

<p>Assuming you are on Windows and viewing the .CSV with Excel or Notepad, use the following line on Python 3.  With only this change (and fixing indentation of your post), You will even be able to view the non-ASCII characters correctly.  Notepad and Excel like a UTF-8 BOM signature at the start of the file, which <code>utf-8-sig</code> provides.</p>

<pre><code>with open('usnwr_schools.csv', 'w', newline='', encoding='utf-8-sig') as f:
</code></pre>

<p>If reading the file in another Python script, make sure to read the file with the following.  Your example of what you read <code>b'University of Michigan\xe2\x80\x94\xe2\x80\x8bAnn Arbor'</code> was read in binary mode <code>'rb'</code>.</p>

<pre><code>with open('usnwr_schools.csv', encoding='utf-8-sig') as f:
</code></pre>

<p>If on Linux, you can use <code>utf8</code> instead of <code>utf-8-sig</code>.</p>

<p>As an aside, you can replace your loops with:</p>

<pre><code>with open('usnwr_schools.csv', 'w', newline='', encoding='utf-8-sig') as f:
    writer = csv.writer(f)
    for school in reqSoup:
        x = reqSoup.find_all("a", {"class" : "school-name"})
        for item in x:
            y = item.get_text()
            writer.writerow([y])
</code></pre>

<p>Reading it back:</p>

<pre><code>with open('usnwr_schools.csv',encoding='utf-8-sig') as f:
    print(f.read())
</code></pre>

<p>Output:</p>

<pre class="lang-none prettyprint-override"><code>Massachusetts Institute of Technology
Stanford University
University of CaliforniaââBerkeley
California Institute of Technology
Carnegie Mellon University
University of MichiganââAnn Arbor
Georgia Institute of Technology
University of IllinoisââUrbana-âChampaign
Purdue UniversityââWest Lafayette
University of TexasââAustin (Cockrell)
Texas A&amp;M; UniversityââCollege Station (Look)
Cornell University
University of Southern California (Viterbi)
Columbia University (Fu Foundation)
University of CaliforniaââLos Angeles (Samueli)
University of CaliforniaââSan Diego (Jacobs)
Princeton University
Northwestern University (McCormick)
University of Pennsylvania
Johns Hopkins University (Whiting)
Virginia Tech
University of CaliforniaââSanta Barbara
Harvard University
University of MarylandââCollege Park (Clark)
University of Washington
Massachusetts Institute of Technology
Stanford University
University of CaliforniaââBerkeley
California Institute of Technology
Carnegie Mellon University
University of MichiganââAnn Arbor
Georgia Institute of Technology
University of IllinoisââUrbana-âChampaign
Purdue UniversityââWest Lafayette
University of TexasââAustin (Cockrell)
Texas A&amp;M; UniversityââCollege Station (Look)
Cornell University
University of Southern California (Viterbi)
Columbia University (Fu Foundation)
University of CaliforniaââLos Angeles (Samueli)
University of CaliforniaââSan Diego (Jacobs)
Princeton University
Northwestern University (McCormick)
University of Pennsylvania
Johns Hopkins University (Whiting)
Virginia Tech
University of CaliforniaââSanta Barbara
Harvard University
University of MarylandââCollege Park (Clark)
University of Washington
Massachusetts Institute of Technology
Stanford University
University of CaliforniaââBerkeley
California Institute of Technology
Carnegie Mellon University
University of MichiganââAnn Arbor
Georgia Institute of Technology
University of IllinoisââUrbana-âChampaign
Purdue UniversityââWest Lafayette
University of TexasââAustin (Cockrell)
Texas A&amp;M; UniversityââCollege Station (Look)
Cornell University
University of Southern California (Viterbi)
Columbia University (Fu Foundation)
University of CaliforniaââLos Angeles (Samueli)
University of CaliforniaââSan Diego (Jacobs)
Princeton University
Northwestern University (McCormick)
University of Pennsylvania
Johns Hopkins University (Whiting)
Virginia Tech
University of CaliforniaââSanta Barbara
Harvard University
University of MarylandââCollege Park (Clark)
University of Washington
Massachusetts Institute of Technology
Stanford University
University of CaliforniaââBerkeley
California Institute of Technology
Carnegie Mellon University
University of MichiganââAnn Arbor
Georgia Institute of Technology
University of IllinoisââUrbana-âChampaign
Purdue UniversityââWest Lafayette
University of TexasââAustin (Cockrell)
Texas A&amp;M; UniversityââCollege Station (Look)
Cornell University
University of Southern California (Viterbi)
Columbia University (Fu Foundation)
University of CaliforniaââLos Angeles (Samueli)
University of CaliforniaââSan Diego (Jacobs)
Princeton University
Northwestern University (McCormick)
University of Pennsylvania
Johns Hopkins University (Whiting)
Virginia Tech
University of CaliforniaââSanta Barbara
Harvard University
University of MarylandââCollege Park (Clark)
University of Washington
Massachusetts Institute of Technology
Stanford University
University of CaliforniaââBerkeley
California Institute of Technology
Carnegie Mellon University
University of MichiganââAnn Arbor
Georgia Institute of Technology
University of IllinoisââUrbana-âChampaign
Purdue UniversityââWest Lafayette
University of TexasââAustin (Cockrell)
Texas A&amp;M; UniversityââCollege Station (Look)
Cornell University
University of Southern California (Viterbi)
Columbia University (Fu Foundation)
University of CaliforniaââLos Angeles (Samueli)
University of CaliforniaââSan Diego (Jacobs)
Princeton University
Northwestern University (McCormick)
University of Pennsylvania
Johns Hopkins University (Whiting)
Virginia Tech
University of CaliforniaââSanta Barbara
Harvard University
University of MarylandââCollege Park (Clark)
University of Washington
</code></pre>

<p>If you still want to be ASCII only, this will do it:</p>

<pre><code>import requests
import bs4
import csv

results = requests.get('http://grad-schools.usnews.rankingsandreviews.com/best-graduate-schools/top-engineering-schools/eng-rankings?int=a74509')

replacements = {ord('\N{EN DASH}'):'-',
                ord('\N{EM DASH}'):'-',
                ord('\N{ZERO WIDTH SPACE}'):None}

reqSoup = bs4.BeautifulSoup(results.text, "html.parser")

with open('usnwr_schools.csv', 'w', newline='', encoding='ascii') as f:
    writer = csv.writer(f)
    for school in reqSoup:
        x = reqSoup.find_all("a", {"class" : "school-name"})
        for item in x:
            y = item.get_text()
            writer.writerow([y.translate(replacements)])

with open('usnwr_schools.csv',encoding='ascii') as f:
    print(f.read())
</code></pre>

python python-3.x unicode