multiprocessing - calling function with different input files
<p>I have a function which reads in a file, compares a record in that file to a record in another file and depending on a rule, appends a record from the file to one of two lists. </p>

<p>I have an empty list for adding matched results to:</p>

<pre><code>match = []
</code></pre>

<p>I have a list <code>restrictions</code> that I want to compare records in a series of files with.</p>

<p>I have a function for reading in the file I wish to see if contains any matches. If there is a match, I append the record to the <code>match</code> list.</p>

<pre><code>def link_match(file):
    links = json.load(file)
    for link in links:
        found = False
        try:
            for other_link in other_links:
                if link['data'] == other_link['data']:
                    match.append(link)
                    found = True
                else:
                    pass
        else:
            print "not found"
</code></pre>

<p>I have numerous files that I wish to compare and I thus wish to use the multiprocessing library. </p>

<p>I create a list of file names to act as function arguments:</p>

<pre><code>list_files=[]
for file in glob.glob("/path/*.json"):
    list_files.append(file)
</code></pre>

<p>I then use the <code>map</code> feature to call the function with the different input files:</p>

<pre><code>if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=6)
    pool.map(link_match,list_files)
    pool.close()
    pool.join()
</code></pre>

<p>CPU use goes through the roof and by adding in a print line to the function loop I can see that matches are being found and the function is behaving correctly. </p>

<p>However, the <code>match</code> results list remains empty. What am I doing wrong?</p>

<p>When multiprocessing, each subprocess gets its own copy of any global variables in the main module defined before the <code>if __name__ == '__main__':</code> statement. This means that the <code>link_match()</code> function in each one of the processes will be accessing a <em>different</em> <code>match</code> list in your code.</p>

<p>One workaround is to use a shared list, which in turn requires a <a href="https://docs.python.org/2/library/multiprocessing.html#multiprocessing.managers.SyncManager" rel="nofollow">SyncManager</a> to synchronize access to the shared resource among the processes (which is created by calling <code>multiprocessing.Manager()</code>). This is then used to create the list to store the results (which I have named <code>matches</code> instead of <code>match</code>) in the code below.</p>

<p>I also had to use <code>functools.partial()</code> to create a single argument callable out of the revised <code>link_match</code> function which now takes two arguments, not one (which is the kind of function <code>pool.map()</code> expects).</p>

<pre><code>from functools import partial
import glob
import multiprocessing

def link_match(matches, file):  # note: added results list argument
    links = json.load(file)
    for link in links:
        try:
            for other_link in other_links:
                if link['data'] == other_link['data']:
                    matches.append(link)
                else:
                    pass
        else:
            print "not found"

if __name__ == '__main__':
    manager = multiprocessing.Manager()  # create SyncManager
    matches = manager.list()  # create a shared list here
    link_matches = partial(link_match, matches)  # create one arg callable to
                                                 # pass to pool.map()
    pool = multiprocessing.Pool(processes=6)
    list_files = glob.glob("/path/*.json")  # only used here
    pool.map(link_matches, list_files)  # apply partial to files list
    pool.close()
    pool.join()
    print(matches)
</code></pre>

python multiprocessing