Efficient way to retrieve and merge returned arrays from multiprocess function call
<pre><code>def get_version_list(env):
  list = []
  #do some intensive work
  return list

if __name__ == '__main__':
  from multiprocessing import Pool

  pool = Pool()

  result1 = pool.apply_async(get_version_list, ['prod'])
  result2 = pool.apply_async(get_version_list, ['uat'])
  #etc, I have six environment to check.

  alist = result1.get()
  blist = result2.get()
</code></pre>

<p>Ideally, I would like to have a list containing my six environments, and loop on that list to call my function for each environment (in parallel) then unite all the returned lists. The united list should not contain a value multiple times.</p>

<p>something like so (I knwo that code does not work, but it's to give an idea of what wish to do)</p>

<pre><code>if __name__ == '__main__':
  from multiprocessing import Pool

  pool = Pool()

  env = ['uat', 'prod', 'lt', 'lt2', 'cert', 'sin']

  for e in env:
      result = pool.apply_async(get_version_list, [e])

  #Merging the lists of the 6 function calls (unique entries)
  list = result.get()
</code></pre>

<p>Is there a simple way to do it?</p>

<p>Use <code>map_async</code> instead of <code>apply_async</code>.</p>

<pre><code>pool = Pool()
env = ['uat', 'prod', 'lt', 'lt2', 'cert', 'sin']
x = pool.map_async(get_version_list, env)
</code></pre>

<p>And now <code>x</code> will be a list of the results.</p>

python multiprocessing python-multiprocessing