MultiProcessing Lock() doesn't work
<p>I have a program that reads some input text files and write all of them into a list called <code>ListOutput</code> which is a shared memory between two processes used in my program (I used two processes so my program runs faster!) I also have a shared memory variable called <code>processedFiles</code> which stores the names of the already read input files by any of the processes so the current process does not read them again!</p>

<ol>
<li><p>To make sure that two processes do not check the  "processedFiles" at the same time (For example in the beginning, it is possible that at the same time they both may come to the conclusion that "processedFiles" is empty so they read the same file), therefore, I added a <code>Lock</code> around the checking part for <code>processedFiles</code> so one process should complete and release the locked part before the other process checking in to that part!</p>

<p>My problem is that the <code>Lock</code> function seems not to work and when I print the current <code>ProcessName</code> inside the lock part, it shows that both processes are inside the lock part. I cannot figure out what is wrong with my code? (See the output below.)</p></li>
<li><p>Since my main program is not only about reading the input text files and printing them in the list, but it has to do a very complicate operation on the input files before printing them into a list, should I use <code>Pool</code> instead of <code>Process</code> and why?</p></li>
</ol>

<p>Code:</p>

<pre><code>import glob
from multiprocessing import Process, Manager
from threading import *
import timeit
import os

os.chdir("files")

def print_content(ProcessName,processedFiles,ListOutput,lock):

   for file in glob.glob("*.txt"):
      newfile=0

      lock.acquire()

      print "\n Current Process:",ProcessName

      if file not in processedFiles:
         print "\n", file, " not in ", processedFiles," for ",ProcessName
         processedFiles.append(file)
         newfile=1#it is a new file

      lock.release()
      #if it is a new file
      if newfile==1:
         f = open(file,"r")
         lines = f.readlines()
         ListOutput.append(lines)
         f.close()

# Create two processes as follows
try:
   manager = Manager()

   processedFiles = manager.list()
   ListOutput = manager.list()

   lock=Lock()
   p1 = Process(target=print_content, args=("Procees-1",processedFiles,ListOutput,lock))
   p2 = Process(target=print_content, args=("Process-2",processedFiles,ListOutput,lock))

   p1.start()
   p2.start()

   p1.join()
   p2.join()

   print "ListOutput",ListOutput

except:
   print "Error: unable to start process"
</code></pre>

<p>I have 4 input files called <code>1.txt</code> (contains "my car"), <code>2.txt</code> (contains "your car"), <code>3.txt</code> (contains "my book"), <code>4.txt</code> (contains "your book").
The output that it shows me changes in different runs. This is the output in one of the runs:</p>

<pre class="lang-none prettyprint-override"><code>Current Process: Procees-1

Current Process: Process-2

1.txt  not in  []  for  Procees-1

Current Process: Procees-1

2.txt  not in
Current Process: Process-2
['1.txt']  for  Procees-1

2.txt  not in  ['1.txt', '2.txt']  for  Process-2

Current Process: Procees-1

3.txt  not in  ['1.txt', '2.txt', '2.txt']  for  Procees-1

Current Process: Process-2

Current Process: Process-2

4.txt  not in
Current Process: Procees-1
['1.txt', '2.txt', '2.txt', '3.txt']  for  Process-2

4.txt  not in  ['1.txt', '2.txt', '2.txt', '3.txt', '4.txt']  for  Procees-1
ListOutput [['my car'], ['your car'], ['your car'], ['my book'], ['your book'], ['your book']]
</code></pre>

<p>Bingo!  Thanks for including the imports.  <em>Now</em> the problem is obvious ;-)</p>

<p>You need to use a <code>multiprocessing.Lock</code> to get a lock that works across processes.  The <code>Lock</code> you're actually using is implicitly obtained along with a mountain of other stuff via your</p>

<pre><code>from threading import *
</code></pre>

<p>A <code>threading.Lock</code> is useless for your purpose:  it has no effect whatsoever <em>across</em> processes; it only provides exclusion among threads <em>within</em> a single process.</p>

<p>It's usually a Bad Idea to use <code>import *</code>, and this is one reason why.  Even if you changed your second import to</p>

<pre><code>from multiprocessing import Process, Manager, Lock
                                              ^^^^
</code></pre>

<p>it wouldn't do you any good, because <code>from threading import *</code> would overwrite the <code>Lock</code> you really want.</p>

python multithreading multiprocessing