Write a list into a file onto HDFS
<p>I am writing a python code to run on a Hadoop Cluster and need to store some intermediate data in a file. Since I want to run the code on the cluster, I want to write the intermediate data into the <code>/tmp</code> directory on HDFS. I will delete the file immediately after I use it for the next steps. How can I do that? </p>

<p>I know I can use <code>subprocess.call()</code> but how do I write the data into the file? The data I want to write is inside a list.</p>

<p>I tried the following syntax:</p>

<pre><code>for item in mylist:
    subprocess.call(["echo '%s' | hadoop fs -put - /tmp/t"%item], shell=True)
</code></pre>

<p>It writes fine but there is a problem here: For the second record onwards, it throws an error <code>/tmp/t</code> already exists.</p>

<p>Is there a way I can do this?</p>

<p>You're facing that error because HDFS cannot append the files when written from shell. You need to create new files each time you want to dump files.</p>

<p>A better way to do this would be to use a python HDFS client to do the dumping for you. I can recommend <a href="https://github.com/spotify/snakebite" rel="nofollow">snakebite</a>, <a href="http://crs4.github.io/pydoop/" rel="nofollow">pydoop</a> and <a href="https://pypi.python.org/pypi/hdfs/" rel="nofollow">hdfs</a> packages. I haven't tried the third one though so I cannot comment on them, but the other two work fine.</p>

python hadoop subprocess