MultiProcessing slower with more processes
<p>I have a program written in python that reads 4 input text files and writes all of them into a list called <code>ListOutput</code> which is a shared memory between 4 processes used in my program (I used 4 processes so my program runs faster!)</p>

<p>I also have a shared memory variable called <code>processedFiles</code> which stores the names of the already read input files by any of the processes so the current process does not read them again (I used lock so processes do not check the existence of a file inside <code>processedFiles</code> at the same time).</p>

<p>When I only use one process my program runs faster(7 milliseconds) &mdash; my computer has 8 cores. Why is this?</p>

<pre><code>import glob
from multiprocessing import Process, Manager,Lock
import timeit
import os

os.chdir("files")
# Define a function for the Processes
def print_content(ProcessName,processedFiles,ListOutput,lock):
   for file in glob.glob("*.txt"):
      newfile=0

      lock.acquire()

      print "\n Current Process:",ProcessName

      if file not in processedFiles:
         print "\n", file, " not in ", processedFiles," for ",ProcessName
         processedFiles.append(file)
         newfile=1 #it is a new file

      lock.release()

      #if it is a new file
      if newfile==1:
         f = open(file,"r")
         lines = f.readlines()
         ListOutput.append(lines)
         f.close()

         #print "%s: %s" % ( ProcessName, time.ctime(time.time()) )

# Create processes as follows
try:
   manager = Manager()
   processedFiles = manager.list()
   ListOutput = manager.list()
   start = timeit.default_timer()

   lock=Lock()
   p1 = Process(target=print_content, args=("Procees-1",processedFiles,ListOutput,lock))
   p2 = Process(target=print_content, args=("Process-2",processedFiles,ListOutput,lock))
   p3 = Process(target=print_content, args=("Process-3",processedFiles,ListOutput,lock))
   p4 = Process(target=print_content, args=("Process-4",processedFiles,ListOutput,lock))

   p1.start()
   p2.start()
   p3.start()
   p4.start()

   p1.join()
   p2.join()
   p3.join()
   p4.join()

   print "ListOutput",ListOutput
   stop = timeit.default_timer()
   print stop - start
except:
   print "Error: unable to start process"
</code></pre>

<p>Just to add to existing answer, there are certain cases where using <code>multiprocessing</code> really adds value and saves time:</p>

<ol>
<li>Your program does <em>N</em> tasks, which are independent of each other.</li>
<li>Your program does extensive heavy mathematical calculations</li>
<li>As a gotcha to second point, the computation time must be significantly larger. Otherwise, the cost of creation of new process will overshadow the advantage of multiprocessing and your parallel-processing designed program will run slower than a sequential version.</li>
<li>In the ideal case, if your program does file I/O, network I/O operations, then don't parallelize your program, unless you have very strong reason to do so.</li>
<li>To add to fourth point, if your requirement demands, then you can think of creating a <code>multiprocess.Pool</code>, which will be performing task for you, as needed and eliminates the overhead of creation and destruction of process every time.</li>
</ol>

<p>Hope it helps you.</p>

python multithreading multiprocessing