multiprocessing - calling function with different input files
<p>I have a function which reads in a file, compares a record in that file to a record in another file and depending on a rule, appends a record from the file to one of two lists. </p>

<p>I have an empty list for adding matched results to:</p>

<pre><code>match = []
</code></pre>

<p>I have a list <code>restrictions</code> that I want to compare records in a series of files with.</p>

<p>I have a function for reading in the file I wish to see if contains any matches. If there is a match, I append the record to the <code>match</code> list.</p>

<pre><code>def link_match(file):
    links = json.load(file)
    for link in links:
        found = False
        try:
            for other_link in other_links:
                if link['data'] == other_link['data']:
                    match.append(link)
                    found = True
                else:
                    pass
        else:
            print "not found"
</code></pre>

<p>I have numerous files that I wish to compare and I thus wish to use the multiprocessing library. </p>

<p>I create a list of file names to act as function arguments:</p>

<pre><code>list_files=[]
for file in glob.glob("/path/*.json"):
    list_files.append(file)
</code></pre>

<p>I then use the <code>map</code> feature to call the function with the different input files:</p>

<pre><code>if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=6)
    pool.map(link_match,list_files)
    pool.close()
    pool.join()
</code></pre>

<p>CPU use goes through the roof and by adding in a print line to the function loop I can see that matches are being found and the function is behaving correctly. </p>

<p>However, the <code>match</code> results list remains empty. What am I doing wrong?</p>

<p>Multiprocessing creates multiple processes. The context of your "match" variable will now be in that child process, not the parent Python process that kicked the processing off.</p>

<p>Try writing the list results out to a file in your function to see what I mean.</p>

python multiprocessing