Encoding/decoding troubleshooting for python CSVs and JSON files
<p>I initially dumped a file which contained a particular sentence using:</p>

<pre><code> with open(labelFile, "wb") as out:
        json.dump(result, out,indent=4)
</code></pre>

<p>This sentence within the JSON looks like:</p>

<pre><code>"-LSB- 97 -RSB- However , the influx of immigrants from mainland China , approximating NUMBER_SLOT per year , is a significant contributor to its population growth \u00c3 cents \u00c2 $ \u00c2 `` a daily quota of 150 Mainland Chinese with family ties in LOCATION_SLOT are granted a `` one way permit '' .", 
</code></pre>

<p>I then proceeded to load this in via:</p>

<pre><code>with open(sys.argv[1]) as sentenceFile:
    sentenceFile = json.loads(sentenceFile.read())
</code></pre>

<p>process it and then write this out to a CSV using:</p>

<pre><code>with open(sys.argv[2], 'wb') as csvfile:
    fieldnames = ['x','y','z'
                  ]
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    for sentence in sentence2locations2values:
         sentence = unicode(sentence['parsedSentence']).encode("utf-8")
         writer.writerow({'x': sentence})
</code></pre>

<p>Which made the sentence in the CSV file opened in Excel for Mac:</p>

<pre><code>-LSB- 97 -RSB- However , the influx of immigrants from mainland China , approximating NUMBER_SLOT per year , is a significant contributor to its population growth ÃÆ cents Ãâ $ Ãâ `` a daily quota of 150 Mainland Chinese with family ties in LOCATION_SLOT are granted a `` one way permit '' .
</code></pre>

<p>I then proceeded to take this from Excel for Macs to Google Sheets, where it is:</p>

<pre><code>-LSB- 97 -RSB- However , the influx of immigrants from mainland China , approximating NUMBER_SLOT per year , is a significant contributor to its population growth Ã cents Ã $ Ã `` a daily quota of 150 Mainland Chinese with family ties in LOCATION_SLOT are granted a `` one way permit '' .
</code></pre>

<p>Note, very slightly different, the <code>Ã</code> has replaced the <code>Ã</code>.</p>

<p>and then labelled it, bringing it back into Excel for Mac at which point it became back to:</p>

<pre><code>-LSB- 97 -RSB- However , the influx of immigrants from mainland China , approximating NUMBER_SLOT per year , is a significant contributor to its population growth Ã cents Ã $ Ã `` a daily quota of 150 Mainland Chinese with family ties in LOCATION_SLOT are granted a `` one way permit '' .
</code></pre>

<p><strong>How do I initially read in the CSV, containing a sentence like</strong>:</p>

<pre><code>-LSB- 97 -RSB- However , the influx of immigrants from mainland China , approximating NUMBER_SLOT per year , is a significant contributor to its population growth ÃÆ cents Ãâ $ Ãâ `` a daily quota of 150 Mainland Chinese with family ties in LOCATION_SLOT are granted a `` one way permit '' .
</code></pre>

<p><strong>to a value which is:</strong></p>

<pre><code>"-LSB- 97 -RSB- However , the influx of immigrants from mainland China , approximating 45,000 per year , is a significant contributor to its population growth \u00c3 cents \u00c2 $ \u00c2 `` a daily quota of 150 Mainland Chinese with family ties in Hong Kong are granted a `` one way permit '' .", 
</code></pre>

<p>So that it matches what was in the original json dump right at the start of this question?</p>

<p><strong>EDIT</strong></p>

<p>I check from this and see that the encoding of <code>\u00c3</code> to <code>Ã</code>, the format in Google sheets, is actually Latin 8.</p>

<p><strong>EDIT</strong></p>

<p>I ran <code>enca</code> and see that the original dumped file is in 7bit ASCII characters, and my CSV is in unicode. So I need to load in as unicode and convert to 7bit ASCII?</p>

<p>I figured out the solution to this. The solution was to decode the CSV file from its original format (identified as <code>UTF-8</code>) and then the sentence becomes the original one. So:</p>

<pre><code>csvfile = open(sys.argv[1], 'r')

fieldnames = ("x","y","z")
reader = csv.DictReader(csvfile, fieldnames)
next(reader)

for i,row in enumerate(reader):
    row['x'] = row['x'].decode("utf-8")
</code></pre>

<p>The very strange thing that happened is that when I edited the CSV file in Excel for Mac and saved, every time it seems to convert to a different encoding. I warn other users about this as it is a huge headache.</p>

python csv encoding decode utf