Running sequential code in hdfs
<p>I am currently learning the "ins and outs" of working with Hadoop. Here's the </p>

<p>current setup: I have sequential code that I use to create .txt files which I will use as the input data for my mappers. I have currently been running this sequential code "preprocess.py" on a local machine and then moving the generated files to the hdfs, but there are many files that are generated and the moving takes much more time then their generation. </p>

<p>I was wondering if, having copied the "preprocess.py" code to the hdfs, there is any way to run it there allowing the generated files to be created on the hdfs instead of requiring the move. </p>

<p>Using </p>

<pre><code>"hdfs dfs -python preprocess.py" 
</code></pre>

<p>returns an "Unknown command" error, so that obviously will not work. Thank you for your time!</p>

<p>It is possible. Just make sure that you push all generated files to single unix location in your python code. Once they are there you can use <code>subprocess</code> module to run to shift the generated file to HDFS. In code it have to wait until file is transferred. Also for making sure that you are not again copying same file consider naming files differently (trying so will give Hadoop error) and delete the file after HDFS transfer is successful.</p>

python hadoop