Invisible unicode characters loaded to DB in python
<p>There are many questions and fixes for this but none seems to work for me.
My problem is I am reading a file with strings and loading each line into DB.</p>

<p>In file it is looking like normal text,while in DB it is read as a unicode space. I tried replacing it with a space and similar options but none worked.</p>

<p>For example in text file the string will be like:</p>

<pre><code>The abruptÂ departure
</code></pre>

<p>After inserted in DB, there it is looking like:</p>

<pre><code>The abruptÃÂ departure
</code></pre>

<p>When I am trying to run query for data in DB, it is looking like:</p>

<pre><code>"The abrupt\xc2\xa0departure"
</code></pre>

<p>I tried the following:</p>

<pre><code>if "\xc2\xa0"  in str: 
     str.replace('\xa0', ' ')
     str.replace('\xc2', ' ')
     print str
</code></pre>

<p>the above code is printing the string like:</p>

<pre><code>The abruptÂ departure
</code></pre>

<p>but while inserting back to DB, it is still the same.</p>

<p>Any help is appreciated.</p>

<p>Try this:</p>

<p>This will remove <code>Unicode</code> character</p>

<pre><code>&gt;&gt;&gt; s = "The abruptÃ departure"
&gt;&gt;&gt; s = s.decode('unicode_escape').encode('ascii','ignore')
&gt;&gt;&gt; s
'The abrupt departure'
</code></pre>

<p>Or,
You can try with replace as you have tried. But you forget to reassign to same variable. </p>

<pre><code>&gt;&gt;&gt; s = "The abruptÃ departure"
&gt;&gt;&gt; s = s.replace('\xc2', '').replace('\xa0','')
&gt;&gt;&gt; s
'The abrupt departure'
</code></pre>

python mysql string unicode replace