How to use Dask to process a file (or files) in multiple stages
<p>I'm processing a large text file in memory in 3 stages (currently not using <code>pandas</code>/<code>dataframes</code>)</p>

<p>This takes one raw data text file and processes it in four stages.</p>

<ul>
<li><p>Stage 1 processes <code>raw.txt</code> and kicks out <code>stage1.txt</code></p></li>
<li><p>Stage 2 processes <code>stage1.txt</code> and kicks out <code>stage2.txt</code></p></li>
<li><p>Stage 3 processes <code>stage2.txt</code> and kicks out <code>results.txt</code></p></li>
</ul>

<p>How should I set a Dask script to work on this locally?
Beyond this, how would can you set it up to work with multiple of <code>raw.txt</code>.
(i.e. raw1, raw2, raw3)</p>

<p>At the moment each stage method does not return anything but writes the next file to a specific file location which the next method knows about.</p>

<pre><code>def stage_1():
    outputFile=r"C:\Data\Processed\stage_1.txt"

    inputFile=r"C:\Data\RawData\rawData.txt"

    f1 = open(outputFile,"w+")
    f2 = open(inputFile,'r')

    #Process input file f2
    #Write results to f1

    f2.close()
    f1.close()

if __name__ == "__main__":
    stage_1()
    stage_2()
    stage_3()
</code></pre>

<p>I suspect you'll run into a few issues.</p>

<h3>Function Purity</h3>

<p>Dask generally assumes that functions are <a href="http://toolz.readthedocs.io/en/latest/purity.html" rel="nofollow">pure</a> rather than rely on side effects.  If you want to use Dask then I recommend that you change your functions so that they return data rather than produce files.</p>

<p>As a hacky workaround you could pass filenames between functions.</p>

<h3>No Parallelism</h3>

<p>The workflow you've described has no intrinsic parallelism.  You can have dask run your functions but it will just run them one after the other.  You would need to think about how to break open your computation a bit so that there are several function calls that could run in parallel.  Dask will not do this thinking for you.</p>

python dask