Lazily create dask dataframe from generator
<p>I want to lazily create a Dask dataframe from a generator, which looks something like:</p>

<pre><code>[parser.read(local_file_name) for local_file_name in repo.download_files())]
</code></pre>

<p>Where both parser.read and repo.download_files return generators (using yield). parser.read yields a dictionary of key-value pairs, which (if I was just using plain pandas) would collect each dictionary in to a list, and then use:</p>

<pre><code>df = pd.DataFrame(parsed_rows)
</code></pre>

<p>What's the best way to create a dask dataframe from this? The reason is that a) I don't know necessarily the number of results returned, and b) I don't know the memory allocation of the machine that it will be deployed on.</p>

<p>Alternatively what should I be doing differently (e.g. maybe create a bunch of dataframes and then put those in to dask instead?)</p>

<p>Thanks.</p>

<p>If you want to use the single-machine Dask scheduler then you'll need to know how many files you have to begin with.  This might be something like the following:</p>

<pre><code>filenames = repo.download_files()
dataframes = [delayed(load)(filename) for filename in filenames]
df = dd.from_delayed(dataframes)
</code></pre>

<p>If you use the <a href="http://distributed.readthedocs.io/en/latest/" rel="nofollow">distributed scheduler</a> you can add new computations on the fly, but this is a bit more advanced.</p>

python pandas dask