What is map_partitions doing?
<p>The dask API says, that map_partition can be used to "apply a Python function on each DataFrame partition." From this description and according to the usual behaviour of "map", I would expect the return value of map_partitions to be (something like) a list whose length equals the number of partitions. Each element of the list should be one of the return values of the function calls.</p>

<p>However, with respect to the following code, I am not sure, what the return value depends on:</p>

<pre><code>#generate example dataframe
pdf = pd.DataFrame(np.random.randint(0,100,size=(100, 4)), columns=list('ABCD'))
ddf = dd.from_pandas(pdf, npartitions=3)

#define helper function for map. VAL is the return value
VAL = pd.Series({'A': 1})
#VAL = pd.DataFrame({'A': [1]}) #other return values used in this example
#VAL = None
#VAL = 1
def helper(x):
    print('function called\n')
    return VAL

#check result
out = ddf.map_partitions(helper).compute()
print(len(out))
</code></pre>

<ul>
<li><code>VAL = pd.Series({'A': 1})</code> causes 4 function calls (probably one to infer the dtype and 3 for the partitions) and an output with len == 3 and the type pd.Series.</li>
<li><code>pd.DataFrame({'A': [1]})</code> results in the same numbers, however the resulting type is pd.DataFrame.</li>
<li><code>VAL = None</code> causes an TypeError ... why? Couldn't a possible use of map_partitions be to <strong>do</strong> something rather than to <strong>return</strong> something?</li>
<li><code>VAL = 1</code> results in only 2 function calls. The result of map_partitions is the integer 1.</li>
</ul>

<p>Therefore, I want to ask some questions:</p>

<ol>
<li>how is the return value of map_partitions determined? </li>
<li>What influences the number of function calls besides the number of partitions / What criteria has a function to fulfil to be called once with each partition? </li>
<li>What should be the return value of a function, that only "does" something, i.e. a procedure? </li>
<li>How should a function be designed, that returns arbitrary objects?</li>
</ol>

<p>The <a href="http://dask.readthedocs.io/en/latest/dataframe-api.html#dask.dataframe.DataFrame.map_partitions" rel="nofollow">Dask DataFrame.map_partitions</a> function returns a new Dask Dataframe or Series, based on the output type of the mapped function.  See the <a href="http://dask.readthedocs.io/en/latest/dataframe-api.html#dask.dataframe.DataFrame.map_partitions" rel="nofollow">API documentation</a> for a thorough explanation.  </p>

<ol>
<li><p>How is the return value of map_partitions determined?</p>

<p>See the API docs referred to above.  </p></li>
<li><p>What influences the number of function calls besides the number of partitions / What criteria has a function to fulfil to be called once with each partition?</p>

<p>You're correct that we're calling it once immediately to guess the dtypes/columns of the output.  You can avoid this by specifying a <code>meta=</code> keyword directly.  Other than that the function is called once per partition.</p></li>
<li><p>What should be the return value of a function, that only "does" something, i.e. a procedure?</p>

<p>You could always return an empty dataframe.  You might also want to consider transforming your dataframe into a sequence of <a href="http://dask.readthedocs.io/en/latest/delayed.html" rel="nofollow">dask.delayed</a> objects, which are typically more often used for ad-hoc computations.</p></li>
<li><p>How should a function be designed, that returns arbitrary objects?</p>

<p>If your function doesn't return series/dataframes then I recommend converting your dataframe to a sequence of <a href="http://dask.readthedocs.io/en/latest/delayed.html" rel="nofollow">dask.delayed</a> objects with the <a href="http://dask.readthedocs.io/en/latest/dataframe-api.html#dask.dataframe.DataFrame.to_delayed" rel="nofollow">DataFrame.to_delayed</a> method.</p></li>
</ol>

python pandas dask