Initiate new processes instead of reusing prior in Python
<p>I am using multiprocessing in Python with:</p>

<pre><code>import multiprocessing as mp
all_arguments = range(0,20)

pool = mp.Pool(processes=7) 
all_items = [pool.apply_async(main_multiprocessing_function, args=(argument_value,)) for argument_value in all_arguments]
for item in all_items:
    item.get()
</code></pre>

<p>In the above, as far as i am aware, after a worker processor finishes, it moves on to the next value. Is there any way instead to force a 'new' worker processor to be initialed from scratch each time, rather than to re-use the old one?</p>

<p>[Specifically, <code>main_multiprocessing_function</code> calls multiple other functions that each using caching to speed up the processing within each task. All those caches are however redundant for the next item to be processed, and thus am interested in a way of resetting everything back to being fresh]. </p>

<p>From the <a href="https://docs.python.org/3.6/library/multiprocessing.html#multiprocessing.pool.Pool" rel="nofollow">docs</a>:</p>

<blockquote>
  <p>maxtasksperchild is the number of tasks a worker process can complete
  before it will exit and be replaced with a fresh worker process, to
  enable unused resources to be freed. The default maxtasksperchild is
  None, which means worker processes will live as long as the pool.</p>
</blockquote>

<p>Just create the pool as </p>

<pre><code>pool = mp.Pool(processes=7, maxtasksperchild=1)
</code></pre>

python python-multiprocessing