Threads not being executed under supervisord
<p>I am working on a basic crawler which crawls 5 websites concurrently using threads.
For each site it creates a new thread. When I run the program from the shell then the output log indicates that all the 5 threads run as expected.
But when I run this program as a <a href="http://supervisord.org/" rel="nofollow">supervisord</a> program then the log indicates that only 2 threads are being run everytime! The log indicates that the all the 5 threads have started but only the same two of them are being executed and the rest get stuck.
 I cannot understand why this inconsistency is happening when it is run from a shell and when it run from supervisor. Is there something I am not taking into account?</p>

<p>Here is the code which creates the threads:</p>

<pre><code>for sid in entries:
    url = entries[sid]
    threading.Thread(target=self.crawl_loop, \
                     args=(sid, url)).start()
</code></pre>

<p><strong>UPDATES:</strong>
As suggested by tdelaney in the comments, I changed the working directory in the supervisord configuration and now all the threads are being run as expected. Though I still don't understand that why setting the working directory to the crawler file directory rectifies the issue. Perhaps some one who knows about how supervisor manages processes can explain?</p>

<p>AFAIK python threads can't do threads properly because it is not thread safe. It just gives you a facility to simulate simultaneous run of the code. Your code will still use 1 core only.</p>

<p><a href="https://wiki.python.org/moin/GlobalInterpreterLock" rel="nofollow">https://wiki.python.org/moin/GlobalInterpreterLock</a></p>

<p><a href="https://en.wikibooks.org/wiki/Python_Programming/Threading" rel="nofollow">https://en.wikibooks.org/wiki/Python_Programming/Threading</a></p>

<p>Therefore it is possible that it does not spawn more processes/threads.</p>

<p>You should use multiprocessing I think?</p>

<p><a href="https://docs.python.org/2/library/multiprocessing.html" rel="nofollow">https://docs.python.org/2/library/multiprocessing.html</a></p>

python multithreading supervisord