Limit Python Threads : Resource Temporarily Unavailable
<p>I use python threading.Thread to spawn threads that execute a small utility for every filename found in os.walk() and get its output. I tried limiting number of threads using:</p>

<pre><code>ThreadLimiter = threading.BoundedSemaphore(3)
</code></pre>

<p>and </p>

<pre><code>ThreadLimiter.acquire() 
</code></pre>

<p>in start of run method and </p>

<pre><code>ThreadLimiter.release() 
</code></pre>

<p>at end of run method</p>

<p>But I still get the below error message when I run the python program. Any suggestions on improving this ?</p>

<pre><code>bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
</code></pre>

<p>When <code>run</code> executes, the Thread has <em>already</em> started. Using a limit inside of <code>run</code> will not limit the number of <em>running</em> threads but of <em>finishing</em> threads - making the problem worse!</p>

<p>Either:</p>

<ul>
<li>Modify <code>start</code> to delay launching the threads.</li>
<li>In your <code>os.walk</code> loop, keep a list of active threads and block using <code>thread.join</code> when there are too many.</li>
<li>Use a thread pool, e.g. <code>multiprocessing.pool.ThreadPool</code>.</li>
</ul>

python multithreading