threading.Lock() performance issues
<p>I have multiple threads:</p>

<pre><code>dispQ = Queue.Queue()
stop_thr_event = threading.Event()

def worker (stop_event):
    while not stop_event.wait(0):
        try:
            job = dispQ.get(timeout=1)
            job.waitcount -= 1
            dispQ.task_done()
        except Queue.Empty, msg:
            continue

# create job objects and put into dispQ here
for j in range(NUM_OF_JOBS):
    j = Job()
    dispQ.put(j)

# NUM_OF_THREADS could be 10-20 ish
running_threads = []
for t in range(NUM_OF_THREADS):
    t1 = threading.Thread( target=worker, args=(stop_thr_event,) )
    t1.daemon = True
    t1.start()
    running_threads.append(t1)


stop_thr_event.set()
for t in running_threads:
    t.join()
</code></pre>

<p>The code above was giving me some very strange behavior.
I've ended up finding out that it was due to decrementing waitcount with out a lock</p>

<p>I 've added an attribute to Job class <code>self.thr_lock = threading.Lock()</code>
Then I've changed it to</p>

<pre><code>with job.thr_lock:
    job.waitcount -= 1
</code></pre>

<p>This seems to fix the strange behavior but it looks like it has degraded in performance.</p>

<p>Is this expected? is there way to optimize locking?<br>
Would it be better to have one global lock rather than one lock per job object?</p>

<p>About the only way to "optimize" threading would be to break the processing down in blocks or chunks of work that can be performed at the same time. This mostly means doing input or output (I/O) because that is the only time the interpreter will release the Global Interpreter Lock, aka the GIL.</p>

<p>In actuality there is often no gain or even a net slow-down when threading is added due to the overhead of using it unless the above condition is met.</p>

<p>It would probably be worse if you used a single global lock for all the shared resources because it would make parts of the program wait when they really didn't need to do so since it wouldn't distinguish what resource was needed so unnecessary waiting would occur.</p>

<p>You might find the PyCon 2015 talk David Beasley gave titled <a href="https://www.youtube.com/watch?v=MCs5OvhV9S4" rel="nofollow"><em>Python Concurrency From the Ground Up</em></a> of interest. It covers threads, event loops, and coroutines.</p>

python multithreading