python 27 - Creating and running instances of another script in parallel
<p>I'm attempting to build a multiprocessing script that retrieves dicts of attributes from a MySQL table and then runs instances of my main script in <strong>parallel</strong>, using each dict retrieved from the MySQL table as an argument to each instance of the main script. The main script has a method called queen_bee() that's responsible for ensuring that all the other methods have the correct information and are executed in the proper order.</p>

<p>I have tried to iterate through the list of dicts in order to create/run parallel processes of the main script using the multiprocessing library. But they end up running consecutively, not concurrently:</p>

<pre><code>from my_main_script import my_main_class as main
import multiprocessing as mp

def create_list_of_attribute_dicts():
    ...
    return list_of_dicts

for each_dict in list_of_dicts:
    instance = main(each_dict)
    p = mp.Process(target=instance.queen_bee(),args=(each_dict,))
    p.start()
    ...
</code></pre>

<p>I have also tried using the multiprocessing library's Pool.map() method. But I can't figure out how to instantiate the main script one time for each dict using Pool.map():</p>

<pre><code>...
pool = mp.Pool()
jobs = pool.map(main.queen_bee(),list_of_dicts)
</code></pre>

<p>The Pool.map method seems to be the cleanest, most pythonic way to get these instances to run in parallel, but I'm hung up on the proper way to do that in this case. I know the above 'jobs' variable will fail because 'main' has not been instantiated. However, I can't figure out how to pass each dict as an argument to separate instances of the main class and then run those instances using the map method. I'm open to trying a different approach. Thanks in advance for your help.</p>

<p>I think I have it licked. The answer appears to have been to use Pool's map function to call a worker function that's <strong>inside</strong> the same multiprocessing script. The worker function then instantiates the main script that calls its own worker function ('queen_bee' in this case).</p>

<p>Using my code example from above:</p>

<pre><code>from my_main_script import my_main_class as main
import multiprocessing as mp

def create_list_of_active_job_attribute_dicts():
    ...
    return active_jobs

def call_the_queen(individual_active_job):
    instance = main(individual_active_job)
    instance.queen_bee()

if __name__ == '__main__':
    pool = mp.Pool(processes = 4)
    pool.map(call_the_queen,active_jobs)
    pool.close()
    pool.join()
</code></pre>

<p>I have no idea <strong>why</strong> this approach worked and the other one didn't. If anybody wants to weigh in and explain, I would be grateful. Thanks @coder for your help!</p>

python parallel-processing multiprocessing