Writing Dask partitions into single file
<p>New to <code>dask</code>,I have a <code>1GB</code> CSV file when I read it in <code>dask</code> dataframe it creates around 50 partitions after my changes in the file when I write, it creates as many files as partitions.<br>
<strong>Is there a way to write all partitions to single CSV file and is there a way access partitions?</strong><br>
Thank you.</p>

<h3>Short answer</h3>

<p>No, Dask.dataframe.to_csv only writes CSV files to different files, one file per partition.  However, there are ways around this.</p>

<h3>Concatenate Afterwards</h3>

<p>Perhaps just concatenate the files after dask.dataframe writes them?  This is likely to be near-optimal in terms of performance.</p>

<pre><code>df.to_csv('/path/to/myfiles.*.csv')
from glob import glob
filenames = glob('/path/to/myfiles.*.csv')
with open('outfile.csv', 'w') as out:
    for fn in filenames:
        with open(fn) as f:
            out.write(f.read())  # maybe add endline here as well?
</code></pre>

<h3>Or use Dask.delayed</h3>

<p>However, you can do this yourself using <a href="http://dask.pydata.org/en/latest/delayed.html" rel="nofollow">dask.delayed</a>, by <a href="http://dask.pydata.org/en/latest/delayed-collections.html" rel="nofollow">using dask.delayed alongside dataframes</a></p>

<p>This gives you a list of delayed values that you can use however you like:</p>

<pre><code>list_of_delayed_values = df.to_delayed()
</code></pre>

<p>It's then up to you to structure a computation to write these partitions sequentially to a single file.  This isn't hard to do, but can cause a bit of backup on the scheduler.</p>

python dask