python: queue.queue() best practice to set size?
<p>so I have Queue.Queue()</p>

<p>i have bunch of producers who puts jobs into that Queue.Queue() and bunch of consumers who pops from the queue</p>

<p>1) is there benefit of capping the Queue size vs. not doing so?<br>
2) by not capping the size, does it really not have any size limit? can grow forever?</p>

<p>I've noticed that deadlock seems to occur more when queue has a fixed size</p>

<p>If you don't cap the size, the <code>Queue</code> can grow until you run out of memory.  So no size limit is imposed by Python, but your machine still has finite resources.</p>

<p>In some applications (probably most), the programmer knows memory consumption can't become a problem, due to the specific character of their application.</p>

<p>But if, e.g., you have producers that "run forever", and consumers that run much slower than producers, capping the size is essential to avoid unbounded memory demands.</p>

<p>As to deadlocks, it's highly unlikely that the implementation of <code>Queue</code> is responsible for a deadlock regardless of whether the <code>Queue</code>'s size is bounded; far more likely that deadlocks are due to flaws in application code.  For example, picture a producer that fetches things over a network, and mistakenly suppresses errors when the network connection is broken.  Then the producer can fail to produce any new items, and so a consumer will eventually block forever waiting for something new to show up on the <code>Queue</code>.  That "looks like" deadlock, but the cause really has nothing to do with <code>Queue</code>.</p>

python multithreading queue