limited number of user-initiated background processes
<p>I need to allow users to submit requests for very, very large jobs. We are talking 100 gigabytes of memory and 20 hours of computing time. This costs our company a lot of money, so it was stipulated that only 2 jobs could be running at any time, and requests for new jobs when 2 are already running would be rejected (and the user notified that the server is busy).</p>

<p>My current solution uses an Executor from concurrent.futures, and requires setting the Apache server to run only one process, reducing responsiveness (current user count is very low, so it's okay for now).</p>

<p>If possible I would like to use Celery for this, but I did not see in the documentation any way to accomplish this particular setting.</p>

<p>How can I run up to a limited number of jobs in the background in a Django application, and notify users when jobs are rejected because the server is busy?</p>

<p><strong>First</strong></p>

<p>You need the first part of <a href="http://stackoverflow.com/a/39447549/4151886">SpiXel's solution</a>. According to him, "<strong>you only create two worker processes with concurrency=1</strong>".</p>

<p><strong>Second</strong></p>

<p>Set the <a href="http://docs.celeryproject.org/en/latest/userguide/workers.html#time-limits" rel="nofollow"><strong>time out</strong></a> for the task waiting in the queue, which is set <a href="http://docs.celeryproject.org/en/latest/configuration.html#celery-event-queue-ttl" rel="nofollow">CELERY_EVENT_QUEUE_TTL</a> and the <strong>queue length limit</strong> according to <a href="http://stackoverflow.com/a/35309730/4151886">how to limit number of tasks in queue and stop feeding when full?</a>. </p>

<p>Therefore, when the two work running jobs, and the task in the queue waiting like 10 sec or any period time you like, the task will be time out. Or if the queue has been fulfilled, new arrival tasks will be dropped out.  </p>

<p><strong>Third</strong></p>

<p>you need extra things to deal with notifying "users when jobs are rejected because the server is busy".</p>

<p><a href="https://www.rabbitmq.com/dlx.html" rel="nofollow">Dead Letter Exchanges</a> is what you need. Every time a task is failed because of the queue length limit or message timeout. "Messages will be dropped or dead-lettered from the front of the queue to make room for new messages once the limit is reached." </p>

<p>You can set <strong>"x-dead-letter-exchange"</strong> to route to another queue, once this queue receive the dead lettered message, you can send a notification message to users.</p>

python django asynchronous celery background-process