threading.Lock() performance issues
<p>I have multiple threads:</p>

<pre><code>dispQ = Queue.Queue()
stop_thr_event = threading.Event()

def worker (stop_event):
    while not stop_event.wait(0):
        try:
            job = dispQ.get(timeout=1)
            job.waitcount -= 1
            dispQ.task_done()
        except Queue.Empty, msg:
            continue

# create job objects and put into dispQ here
for j in range(NUM_OF_JOBS):
    j = Job()
    dispQ.put(j)

# NUM_OF_THREADS could be 10-20 ish
running_threads = []
for t in range(NUM_OF_THREADS):
    t1 = threading.Thread( target=worker, args=(stop_thr_event,) )
    t1.daemon = True
    t1.start()
    running_threads.append(t1)


stop_thr_event.set()
for t in running_threads:
    t.join()
</code></pre>

<p>The code above was giving me some very strange behavior.
I've ended up finding out that it was due to decrementing waitcount with out a lock</p>

<p>I 've added an attribute to Job class <code>self.thr_lock = threading.Lock()</code>
Then I've changed it to</p>

<pre><code>with job.thr_lock:
    job.waitcount -= 1
</code></pre>

<p>This seems to fix the strange behavior but it looks like it has degraded in performance.</p>

<p>Is this expected? is there way to optimize locking?<br>
Would it be better to have one global lock rather than one lock per job object?</p>

<p>It's hard to answer your question based on your code. Locks do have some inherent cost, nothing is free, but normally it is quite small. If your jobs are very small, you might want to consider "chunking" them, that way you have many fewer acquire/release calls relative to the amount of work being done by each thread.</p>

<p>A related but separate issue is one of threads blocking each other. You might notice large performance issues if many threads are waiting on the same lock(s). Here your threads are sitting idle waiting on each other. In some cases this cannot be avoided because there is a shared resource which is a performance bottlenecking. In other cases you can re-organize your code to avoid this performance penalty.</p>

<p>There are some things in your example code that make me thing that it might be very different from actual application. First, your example code doesn't share job objects between threads. If you're not sharing job objects you shouldn't need locks on them. Second, as written your example code might not empty the queue before finishing. It will exit as soon as you hit <code>stop_thr_event.set()</code> leaving any remaining jobs in queue, is this by design?</p>

python multithreading