limited number of user-initiated background processes
<p>I need to allow users to submit requests for very, very large jobs. We are talking 100 gigabytes of memory and 20 hours of computing time. This costs our company a lot of money, so it was stipulated that only 2 jobs could be running at any time, and requests for new jobs when 2 are already running would be rejected (and the user notified that the server is busy).</p>

<p>My current solution uses an Executor from concurrent.futures, and requires setting the Apache server to run only one process, reducing responsiveness (current user count is very low, so it's okay for now).</p>

<p>If possible I would like to use Celery for this, but I did not see in the documentation any way to accomplish this particular setting.</p>

<p>How can I run up to a limited number of jobs in the background in a Django application, and notify users when jobs are rejected because the server is busy?</p>

<p>First of all you need to limit concurrency on your worker (<a href="http://docs.celeryproject.org/en/latest/userguide/workers.html#starting-the-worker" rel="nofollow">docs</a>):</p>

<pre><code>celery -A proj worker --loglevel=INFO --concurrency=2 -n &lt;worker_name&gt;
</code></pre>

<p>This will help to make sure that you do not have more than 2 active tasks even if you will have errors in the code.</p>

<p>Now you have 2 ways to implement task number validation:</p>

<ol>
<li><p>You can use <a href="http://docs.celeryproject.org/en/latest/_modules/celery/app/control.html#Inspect" rel="nofollow">inspect</a> to get number of active and scheduled tasks:</p>

<pre><code> from celery import current_app

 def start_job():
      inspect = current_app.control.inspect()
      active_tasks = inspect.active() or {}
      scheduled_tasks = inspect.scheduled() or {}
      worker_key = 'celery@%s' % &lt;worker_name&gt;
      worker_tasks = active_tasks.get(worker_key, []) + scheduled_tasks.get(worker_key, [])
      if len(worker_tasks) &gt;= 2:
          raise MyCustomException('It is impossible to start more than 2 tasks.') 
      else:
          my_task.delay()
</code></pre></li>
<li><p>You can store number of currently executing tasks in DB and validate task execution based on it.</p></li>
</ol>

<p>Second approach could be better if you want to scale your functionality - introduce premium users or do not allow to execute 2 requests by one user.  </p>

python django asynchronous celery background-process