How to find why a task fails in dask distributed?
<p>I am developing a distributed computing system using <code>dask.distributed</code>. Tasks that I submit to it with the <code>Executor.map</code> function sometimes fail, while others seeming identical, run successfully. </p>

<p>Does the framework provide any means to diagnose problems?</p>

<p><strong>update</strong> 
By failing I mean increasing counter of failed tasks in the Bokeh web UI, provided by the scheduler. Counter of finished tasks increases too.</p>

<p>Function that is run by the <code>Executor.map</code> returns <code>None</code>. It communicates to a database, retrieves some rows  from its table, performs calculations and updates values. </p>

<p>I've got more than 40000 tasks in map, so it is a bit tedious to study logs.</p>

<p>If a task fails then any attempt to retrieve the result will raise the same error that occurred on the worker</p>

<pre><code>In [1]: from distributed import Client

In [2]: c = Client()

In [3]: def div(x, y):
   ...:     return x / y
   ...: 

In [4]: future = c.submit(div, 1, 0)

In [5]: future.result()
&lt;ipython-input-3-398a43a7781e&gt; in div()
      1 def div(x, y):
----&gt; 2     return x / y

ZeroDivisionError: division by zero
</code></pre>

<p>However, other things can go wrong.  For example you might not have the same software on your workers as on your client or your network might not let connections go through, or any of the other things that happen in real-world networks.  To help diagnose these there are a few options:</p>

<ol>
<li>You can use the <a href="http://distributed.readthedocs.io/en/latest/web.html" rel="nofollow">web interface</a> to track the progress of your tasks and workers</li>
<li>You can <a href="http://distributed.readthedocs.io/en/latest/ipython.html#launch-ipython-within-dask-workers" rel="nofollow">start IPython kernels</a> in the scheduler or workers to inspect them directly</li>
</ol>

python distributed dask