How to apply multiprocessing technique in python for-loop?
<p>I have a long list of user(about 200,000) and a corresponding data frame <code>df</code> with their attributes. Now I'd like to write a for loop to measure pair-wise similarity of the users. The code is following:</p>

<pre><code>df2record = pd.DataFrame(columns=['u1', 'u2', 'sim'])
for u1 in reversed(user_list):
    for u2 in reversed(list(range(1, u1))):
        sim = measure_sim(df[u1], df[u2]))
        if sim &lt; 0.6:
            continue
        else:
            df2record = df2record.append(pd.Series([u1, u2, sim], index=['u1', 'u2', 'sim']), ignore_index=True)
</code></pre>

<p>Now I wanna run this for loop with multiprocessing and I have read some tutorial. But I still have no idea to handle it properly. Seems that I should set reasonable number of processes first, like <code>6</code>. And then I should feed each loop into one process. But the problem is how can I know the task in a certain process has been done so that a new loop can begin? Could you help me with this? Thanks you in advance!</p>

<p>You can use <a href="https://docs.python.org/3.5/library/multiprocessing.html#multiprocessing.pool.Pool" rel="nofollow">multiprocessing.Pool</a> which provides method <code>map</code> that maps pool of processes over given iterable. Here's some example code:</p>

<pre><code>def pairGen():
    for u1 in reversed(user_list):
        for u2 in reversed(list(range(1, u1))):
            yield (u1, u2)

def processFun(pair):
    u1, u2 = pair
    sim = measure_sim(df[u1], df[u2]))
    if sim &lt; 0.6:
        return None
    else:
        return pd.Series([u1, u2, sim], index=['u1', 'u2', 'sim'])

def main():
    with multiprocessing.Pool(processes=6) as pool:
       vals = pool.map(processFun, pairGen())

    df2record = pd.DataFrame(columns=['u1', 'u2', 'sim'])
    for v in vals:
       if vals != None:
           df2record = df2record.append(v, ignore_index=True)
</code></pre>

python multiprocessing