How to map results of `dask.DataFrame` to csvs
<p>I create a dataframe with <code>df=dask.DataFrame.read_csv('s3://bucket/*.csv')</code>. When i execute a <code>df[df.a.isnull()].compute</code> operation, i get a set of rows returned that match the filter criteria. I would like to know which files do these returned rows belong in so that i could investigate why such records  have null values. The <code>DataFrame</code> has billions of rows and the records with the missing values are in single digits. Is there an efficient way to do so?</p>

<p>If your CSV files are small then I recommend creating one partition per file</p>

<pre><code>df = dd.read_csv('s3://bucket/*.csv', blocksize=None)
</code></pre>

<p>And then computing the number of null elements per partition:</p>

<pre><code>counts = df.a.isnull().map_partitions(sum).compute()
</code></pre>

<p>You could then find the filenames</p>

<pre><code>from s3fs import S3FileSystem
s3 = S3FileSystem()
filenames = s3.glob('s3://bucket/*.csv')
</code></pre>

<p>And compare the two</p>

<pre><code>dict(zip(filenames, counts))
</code></pre>

python dask