Using Python's multiprocessing to calculate the sum of integers from one long input line
<p>I want to use Python's multiprocessing module for the following:
Map an input line to a list of integers and calculate the sum of this list.</p>

<p>The input line is initially a string where the items to be summed are separated by spaces.</p>

<p>What I have tried is this:</p>

<pre><code>from itertools import imap

my_input = '1000000000 ' * int(1e6)
print sum(imap(int, my_input.split()))
</code></pre>

<p>This takes about 600ms on my machine, but I would like to make it faster with multiprocessing.</p>

<p>It seems that the bottleneck is in the mapping-part since the sum-method is pretty fast when applied to a ready list of integers:</p>

<pre><code>&gt;&gt;&gt; int_list = [int(1e9)] * int(1e6)
&gt;&gt;&gt; %time sum(int_list)
CPU times: user 7.38 ms, sys: 5 Âµs, total: 7.38 ms
Wall time: 7.4 ms
&gt;&gt;&gt; 1000000000000000
</code></pre>

<p>I tried to apply the instructions from <a href="http://stackoverflow.com/questions/29785427/how-to-parallel-sum-a-loop-using-multiprocessing-in-python">this question</a> but as I'm quite new to using multiprocessing, I couldn't fit the instructions to this problem.</p>

<p>Using a feature of Unix systems called <a href="https://en.wikipedia.org/wiki/Fork_(system_call)" rel="nofollow">forking</a>, you can read (not write) data from the parent process with zero overhead. Normally, you would have to copy the data over, but forking a process in Unix allows you to circumvent this.</p>

<p>Using this, the job in the pool can access the whole input string and extract the part that it will work on. It can then split and parse this section of the string on its own and return the sum of the integers in its section. </p>

<pre><code>from multiprocessing import Pool, cpu_count
from time import time


def serial(data):
    return sum(map(int, data.split()))


def parallel(data):
    processes = cpu_count()

    with Pool(processes) as pool:
        args = zip(
            ["input_"] * processes, # name of global to access
            range(processes), # job number
            [processes] * processes # total number of jobs 
        )
        return sum(pool.map(job, args, chunksize=1))


def job(args):
    global_name, job_number, total_jobs = args
    data = globals()[global_name]
    chunk = get_chunk(data, job_number, total_jobs)

    return serial(chunk)


def get_chunk(string, job_number, total_jobs):
    """This function may mess up if the number of integers in each chunk is low (1-2).
    It also assumes there is only 1 space separating integers."""
    approx_chunk_size = len(string) // total_jobs

    # initial estimates
    start = approx_chunk_size * job_number
    end = start + approx_chunk_size

    if start and not string.startswith(" ", start - 1):
        # if string[start] is not beginning of a number, advance to start of next number
        start = string.index(" ", start) + 1

    if job_number == total_jobs:
        # last job
        end = None
    elif not string.startswith(" ", end - 1):
        # if string[end] is part of a number, then advance to end of number
        end = string.index(" ", end - 1)

    return string[start:end]


def timeit(func, *args, **kwargs):
    "Simple timing function"
    start = time()
    result = func(*args, **kwargs)
    end = time()
    print("{} took {} seconds".format(func.__name__, end - start))
    return result


if __name__ == "__main__":
#    from multiprocessing.dummy import Pool # uncomment this for testing

    input_ = "1000000000 " * int(1e6)

    actual = timeit(parallel, input_)
    expected = timeit(serial, input_)
    assert actual == expected
</code></pre>

python python-2.7 python-multiprocessing