save the file in HDFS from a pair RDD
<p>Below is the python script which I am using to write in HDFS. RDD is a pair RDD.The script works fine however it creates an entry as tuple in HDFS.Is is possible to remove the tuple and just create comma separated entries in HDFS.</p>

<pre><code>    import sys
from pyspark import SparkContext

if len(sys.argv) &lt; 2:
    print 'Insufficient arguments'
    sys.exit()

sc = SparkContext()
initialrdd1 = sc.textFile(sys.argv[1])
finalRDD1 = initialrdd1.map(lambda x:x.split(',')).map(lambda x :(x[1],x[0])).sortByKey()
print finalRDD1.getNumPartitions()
finalRDD1.saveAsTextFile('/export_dir/result3/')
</code></pre>

<p>file storing in HDFS is in below format</p>

<pre><code>(u'Alpha', u'E03')
(u'Beta', u'E02')
(u'Gamma', u'E05')
(u'Delta', u'E09')
</code></pre>

<pre><code>finalRDD1 = initialrdd1.map(lambda x:x.split(',')).map(lambda x :(x[1],x[0])).sortByKey()
</code></pre>

<p>Understand your code. In your initial RDD, you are mapping each entry to a tuple. <strong>map(lambda x :(x[1],x[0]))</strong></p>

<pre><code>finalRDD1.saveAsTextFile('/export_dir/result3/')
</code></pre>

<p>After the sortByKey operation, you directly proceed to save the RDD as textfile. </p>

<p>In order to save entries as CSV, you must to specify it explicitly like so - </p>

<pre><code>def csv_format(data):
    return ','.join(str(d) for d in data)

# Rest of the code ...

finalRDD1.map(csv_format).saveAsTextFile('/export_dir/result3/')
</code></pre>

python apache-spark hdfs pyspark