Python: Deadlock of a single lock in multiprocessing
<p>I'm using pyserial to acquire data with multiprocessing. The way I share data is very simple. So:</p>

<p>I have member objects in my class:</p>

<pre><code>self.mpManager = mp.Manager()
self.shared_return_list = self.mpManager.list()
self.shared_result_lock = mp.Lock()
</code></pre>

<p>I call my multiprocessing process this way:</p>

<pre><code>process = mp.Process(target=do_my_stuff, 
args=(self.shared_stopped, self.shared_return_list, self.shared_result_lock)
)
</code></pre>

<p>where <code>do_my_stuff</code> is a global function.</p>

<p>Now The part the fills the list in the process function:</p>

<pre><code>if len(acqBuffer) &gt; acquisitionSpecs["LengthToPass"]:
    shared_lock.acquire()
    shared_return_list.extend(acqBuffer)
    del acqBuffer[:]
    shared_lock.release()
</code></pre>

<p>And the part that takes that to the local thread for use is:</p>

<pre><code>while len(self.acqBuffer) &lt;= 0 and (not self.stopped):
    #copy list from shared buffer and empty it
    self.shared_result_lock.acquire()
    self.acqBuffer.extend(self.shared_return_list)
    del self.shared_return_list[:]
    self.shared_result_lock.release()
</code></pre>

<p><strong>The problem</strong>:</p>

<p>Although there's only 1 lock, my program is occasionally ending in a deadlock somehow! After waiting some time, my program freezes. After adding prints before and after the locks, I found that it freezes at a lock and reaches a deadlock somehow.</p>

<p>If I use a recursive lock, <code>RLock()</code>, it works with no problems. Not sure whether I should do that.</p>

<p>How is this possible? Am I doing something wrong? I expect if both processes try to acquire the lock, they should block until the other process unlocks the lock.</p>

<p>Without having a <a href="http://sscce.org/" rel="nofollow">SSCCE</a>, it's difficult to know if there's something else going on in your code or not.</p>

<p>One possibility is that there is an exception thrown after a lock is acquired.  Try wrapping each of your locked sections in a try/finally clause.  Eg.</p>

<pre><code>try:
    shared_lock.acquire()
    shared_return_list.extend(acqBuffer)
    del acqBuffer[:]
finally:
    shared_lock.release()
</code></pre>

<p>and:</p>

<pre><code>try:
    self.shared_result_lock.acquire()
    self.acqBuffer.extend(self.shared_return_list)
    del self.shared_return_list[:]
finally:
    self.shared_result_lock.release()
</code></pre>

<p>You could even add exception clauses, and log any exceptions raised, if this turns out to be the issue.</p>

python thread-safety multiprocessing mutex python-3.4