Responsive IPython notebook with running progress bar of dask/distributed
<p>I am running a cluster with <code>dask.distributed</code>. Currently I submit tasks to the cluster with Jupyter notebook that I use as a GUI.</p>

<p>The respective notebook cell contains the following code.</p>

<pre><code>%pylab inline
%load_ext autoreload
%autoreload 2

from distributed import progress
sys.path.append('/path/to/my/python/modules/on/NAS')

import jobs
jobid = jobs.add_new_job(...)
r = jobs.start_job(jobid)
progress(r)   
</code></pre>

<p><code>jobs</code> is the name of my python module. <code>jobs.add_new_job</code> returns a string with job identifier. <code>jobs.start_job</code> returns a list of <code>distributed.client.Future</code>s. Final result of the job is a report with some numbers and some plots in PDF.</p>

<p>Now I'd like to implement a job queue with some indication on what is being processed now and what is waiting.</p>

<p>My goal is to implement the following scenario. </p>

<p>Member of my team prepares some data for a new job, then opens Jupyter notebook in his browser, enters job parameters in the cell in the call to <code>add_new_job</code>, then executes this cell, then closes that page and waits till computations are done. He could also leave the page open and observe the progress.</p>

<p>Up to now I've found that if I submit a single job to a cluster by running the cell once and waiting till everything is done, everything works like a charm.</p>

<p>If I try to submit one more job by simply editing the cell code and running it again, then the cluster stops calculating first submitted job. My explanation of this is that <code>r</code> is deleted and its destructor sends cancellation requests to the cluster.</p>

<p>If I try to submit a new job by making a copy of the notebook, a new empty page opens in the browser, and then it takes very long time until the notebook loads and allows a user to do anything.</p>

<p>Also, progress bar, displayed by <code>progress</code>, very often disappears by itself.</p>

<p>I've read already about JupyterHub, but currently it seems to me that using it is like a shooting sparrows with a heavy artillery, and there should be more simple way.</p>

<blockquote>
  <p>My explanation of this is that r is deleted and its destructor sends cancellation requests to the cluster</p>
</blockquote>

<p>This is correct.  A simple way to avoid this would be to add <code>r</code> to some result set that is not deleted every time you run your cell</p>

<pre><code>-- cell 1 -- 

results = []

-- cell 2 --

import jobs
jobid = jobs.add_new_job(...)
r = jobs.start_job(jobid)
results.append(r)
progress(r)
</code></pre>

python distributed jupyter-notebook dask