Python: Replacing character error with read in file
<p>Goal: I just want to take the comma away as that is the only character that will screw up my (course required) file parsing for bayesian analysis (i.e word,2,4) instead of say (i.e. word,,2,4)</p>

<p>So I'm currently trying to read in an email in the form of a text file from the Enron public corpus online and building a bayesian spam filter.</p>

<p>I've noticed that reading in some of the files are raising errors when trying to manipulate the strings that are present. I am fully aware that some of theses files contain viruses so the encoding of some of the characters might not be valid. However, I'm trying to simply replace a comma within a string and I'm getting the following error:</p>

<p>UnicodeDecodeError: 'ascii' codec can't decode byte 0xc1 in position 1169: ordinal not in range(128)</p>

<p>I have tried everything that this forum has to offer and i've searched everywhere for a solution such as:</p>

<pre><code>with open(file+file_path_stings[i],'r') as filehandle:
        words = str(filehandle.read())
        words = words.replace(',','')
        words = words.split()
</code></pre>

<p>I've also tried many regex attempts... this is one of the versions:</p>

<pre><code>with open(file+file_path_stings[i],'r') as filehandle:
        words = str(filehandle.read())
        words = re.sub(',','',words)
        words = words.split()
</code></pre>

<p>Now, I can simply just regex a version that only lets A-Za-z through but I'm noticing that spam accuracy is heavily being affected by the fact that a lot of the spam files have such special characters.</p>

<p>Any suggestion would be most appreciated. Thanks.</p>

<p>-Robert</p>

<p>So I ended up using another implementation that I found useful as well. It turns out, for some reason, python retains any prior information it had for any previous strings that were originally present. So i've learned its always a good idea to just re-assign it to a different(new) variable as follows:</p>

<pre><code>with open(file+file_path_stings[i],'r') as filehandle:
    words = str(filehandle.read()).split()
    new_array = []
    for word in words:
        new_array.append(word.replace(',','').lower())
    return new_array
</code></pre>

<p>Its a little more expensive as far as storing and assigning data to a whole other variable. However, I've noticed its a lot safer in terms of your string not getting casted to a unicode string. The original problem was this output</p>

<pre><code> print words

 [u'hello,',u'what?',u'is',u'going',u'on?']
</code></pre>

<p>The comma in 'hello' would not get replaced. With the code above you're guaranteed that the comma will be stripped from each word and not casted into a unicode string</p>

<pre><code>print new_array
['hello','what?',u'is',u'going',u'on?']
</code></pre>

<p>As far as performance of the code goes, I'm still training massive files at a decent speed. So it should effect you that much. </p>

<p>Hope this helps!</p>

<p>-Robert</p>

python spam bayesian