Invisible unicode characters loaded to DB in python
<p>There are many questions and fixes for this but none seems to work for me.
My problem is I am reading a file with strings and loading each line into DB.</p>

<p>In file it is looking like normal text,while in DB it is read as a unicode space. I tried replacing it with a space and similar options but none worked.</p>

<p>For example in text file the string will be like:</p>

<pre><code>The abruptÂ departure
</code></pre>

<p>After inserted in DB, there it is looking like:</p>

<pre><code>The abruptÃÂ departure
</code></pre>

<p>When I am trying to run query for data in DB, it is looking like:</p>

<pre><code>"The abrupt\xc2\xa0departure"
</code></pre>

<p>I tried the following:</p>

<pre><code>if "\xc2\xa0"  in str: 
     str.replace('\xa0', ' ')
     str.replace('\xc2', ' ')
     print str
</code></pre>

<p>the above code is printing the string like:</p>

<pre><code>The abruptÂ departure
</code></pre>

<p>but while inserting back to DB, it is still the same.</p>

<p>Any help is appreciated.</p>

<p><code>C2A0</code> is a "NO-BREAK SPACE".  <code>'Ã '</code> is what you see if your <code>CHARATER SET</code> settings are inconsistent.</p>

<p>Doing a <code>replace()</code> is merely masking the problem, and not helping when a different funny character comes into your table.</p>

<p>Since you have not provided enough info to say what you have done correctly versus incorrectly, let me point you at two references:</p>

<ul>
<li><p><a href="http://mysql.rjweb.org/doc.php/charcoll#python" rel="nofollow">Things to check for in Python</a></p></li>
<li><p><a href="http://stackoverflow.com/a/38363567/1766831">The things you <em>should</em> do for utf8, and what may have gone wrong to get "Mojibake"</a></p></li>
</ul>

python mysql string unicode replace