How to map a column with dask
<p>I want to apply a mapping on a DataFrame column. With Pandas this is straight forward:</p>

<pre><code>df["infos"] = df2["numbers"].map(lambda nr: custom_map(nr, hashmap))
</code></pre>

<p>This writes the <code>infos</code> column, based on the <code>custom_map</code> function, and uses the rows in numbers for the <code>lambda</code> statement.</p>

<p>With <a href="http://dask.pydata.org/en/latest/dataframe-api.html" rel="nofollow">dask</a> this isn't that simple. <code>ddf</code> is a dask DataFrame. <code>map_partitions</code> is the equivalent to parallel execution of the mapping on a part of the DataFrame.</p>

<p>This does <strong>not</strong> work because you don't define columns like that in dask.</p>

<pre><code>ddf["infos"] = ddf2["numbers"].map_partitions(lambda nr: custom_map(nr, hashmap))
</code></pre>

<p>Does anyone know how I can use columns here? I don't understand their <a href="http://dask.pydata.org/en/latest/dataframe-api.html#dask.dataframe.DataFrame.map_partitions" rel="nofollow">API documentation</a> at all. </p>

<p>You can use the <a href="http://dask.pydata.org/en/latest/dataframe-api.html#dask.dataframe.Series.map" rel="nofollow">.map</a> method, exactly as in Pandas</p>

<pre><code>In [1]: import dask.dataframe as dd

In [2]: import pandas as pd

In [3]: df = pd.DataFrame({'x': [1, 2, 3]})

In [4]: ddf = dd.from_pandas(df, npartitions=2)

In [5]: df.x.map(lambda x: x + 1)
Out[5]: 
0    2
1    3
2    4
Name: x, dtype: int64

In [6]: ddf.x.map(lambda x: x + 1).compute()
Out[6]: 
0    2
1    3
2    4
Name: x, dtype: int64
</code></pre>

<h3>Metadata</h3>

<p>You may be asked to provide a <code>meta=</code> keyword.  This lets dask.dataframe know the output name and type of your function.  Copying the docstring from <code>map_partitions</code> here:</p>

<pre><code>meta : pd.DataFrame, pd.Series, dict, iterable, tuple, optional

An empty pd.DataFrame or pd.Series that matches the dtypes and 
column names of the output. This metadata is necessary for many 
algorithms in dask dataframe to work. For ease of use, some 
alternative inputs are also available. Instead of a DataFrame, 
a dict of {name: dtype} or iterable of (name, dtype) can be 
provided. Instead of a series, a tuple of (name, dtype) can be 
used. If not provided, dask will try to infer the metadata. 
This may lead to unexpected results, so providing meta is  
recommended. 

For more information, see dask.dataframe.utils.make_meta.
</code></pre>

<p>So in the example above, where my output will be a series with name <code>'x'</code> and dtype <code>int</code> I can do either of the following to be more explicit</p>

<pre><code>&gt;&gt;&gt; ddf.x.map(lambda x: x + 1, meta=('x', int))
</code></pre>

<p>or </p>

<pre><code>&gt;&gt;&gt; ddf.x.map(lambda x: x + 1, meta=pd.Series([], dtype=int, name='x'))
</code></pre>

<p>This tells dask.dataframe what to expect from our function.  If no meta is given then dask.dataframe will try running your function on a little piece of data.  It will raise an error asking for help if this fails.</p>

python pandas dask