MultiProcessing slower with more processes
<p>I have a program written in python that reads 4 input text files and writes all of them into a list called <code>ListOutput</code> which is a shared memory between 4 processes used in my program (I used 4 processes so my program runs faster!)</p>

<p>I also have a shared memory variable called <code>processedFiles</code> which stores the names of the already read input files by any of the processes so the current process does not read them again (I used lock so processes do not check the existence of a file inside <code>processedFiles</code> at the same time).</p>

<p>When I only use one process my program runs faster(7 milliseconds) &mdash; my computer has 8 cores. Why is this?</p>

<pre><code>import glob
from multiprocessing import Process, Manager,Lock
import timeit
import os

os.chdir("files")
# Define a function for the Processes
def print_content(ProcessName,processedFiles,ListOutput,lock):
   for file in glob.glob("*.txt"):
      newfile=0

      lock.acquire()

      print "\n Current Process:",ProcessName

      if file not in processedFiles:
         print "\n", file, " not in ", processedFiles," for ",ProcessName
         processedFiles.append(file)
         newfile=1 #it is a new file

      lock.release()

      #if it is a new file
      if newfile==1:
         f = open(file,"r")
         lines = f.readlines()
         ListOutput.append(lines)
         f.close()

         #print "%s: %s" % ( ProcessName, time.ctime(time.time()) )

# Create processes as follows
try:
   manager = Manager()
   processedFiles = manager.list()
   ListOutput = manager.list()
   start = timeit.default_timer()

   lock=Lock()
   p1 = Process(target=print_content, args=("Procees-1",processedFiles,ListOutput,lock))
   p2 = Process(target=print_content, args=("Process-2",processedFiles,ListOutput,lock))
   p3 = Process(target=print_content, args=("Process-3",processedFiles,ListOutput,lock))
   p4 = Process(target=print_content, args=("Process-4",processedFiles,ListOutput,lock))

   p1.start()
   p2.start()
   p3.start()
   p4.start()

   p1.join()
   p2.join()
   p3.join()
   p4.join()

   print "ListOutput",ListOutput
   stop = timeit.default_timer()
   print stop - start
except:
   print "Error: unable to start process"
</code></pre>

<p>The problem is that what <em>looks</em> like multiprocessing often isn't. Just using more cores doesn't mean doing more work.</p>

<p>The most glaring problem is that you synchronize everything. Selecting files is sequential because you lock, so there is zero gain here. While you are reading in parallel, every line read is written to a shared data structure - which will internally synchronize itself. So the only gain you <em>potentially</em> get is from reading in parallel. Depending on your media, e.g. an HDD instead of an SSD, the sum of multiple readers is actually slower than a single one.</p>

<p>On top of that is the overhead from managing all those processes. Each one needs to be started. Each one needs to be passed its input. Each one must communicate with the others, which happens for practically every action. And don't be fooled, a <code>Manager</code> is nifty but heavyweight.</p>

<p>So aside from gaining little, you add an additional cost. Since you start out with a very small runtime of just <code>7ms</code>, that additional cost can be pretty significant.</p>

<p>In general, <code>multiprocessing</code> is only worth it if you are CPU-bound. That is, your CPU efficiency is close to 100%, i.e. there's more work than what can be done. Generally, this happens when you do lots of computation. Usually, doing mostly I/O is a good indicator that you are <em>not</em> CPU-bound.</p>

python multithreading multiprocessing