PySpark how to read file having string with multiple encoding
<p>I am writing a python spark utility to read files and do some transformation.
File has large amount of data ( upto 12GB ). I use sc.textFile to create a RDD and logic is to pass each line from RDD to a map function which in turn split's the line by "," and run some data transformation( changing fields value based on a mapping ).</p>

<blockquote>
  <p>Sample line from the file.
  0014164,02,031270,09,1,,0,0,0000000000,134314,Mobile,ce87862158eb0dff3023e16850f0417a-cs31,584e2cd63057b7ed,PrivÃ©,Gossip</p>
</blockquote>

<p>Due to values "PrivÃ©" I get UnicodeDecodeError. I tried to following to parse this value:</p>

<pre><code>if isinstance(v[12],basestring):
            v[12] = v[12].encode('utf8')
        else:
            v[12] = unicode(v[12]).encode('utf8')
</code></pre>

<p>but when I write data back to file this field gets translated as 'PrivÃ¯Â¿Â½'.
on Linux source file type is shown as "ISO-8859 text, with very long lines, with CRLF line terminators".</p>

<p>Could someone let me know right way in Spark to read/write files with mixed encoding please.</p>

<p>You can set <code>use_unicode</code> to <code>False</code> when calling <code>textFile</code>. It will give you RDD of <code>str</code> objects (Python 2.x) or <code>bytes</code> objects (Python 3.x) which can further processed using desired encoding.</p>

<pre><code>sc.textFile(path, use_unicode=False).map(lambda x: x.decode("iso-8859-1"))
</code></pre>

python apache-spark pyspark