limited number of user-initiated background processes
<p>I need to allow users to submit requests for very, very large jobs. We are talking 100 gigabytes of memory and 20 hours of computing time. This costs our company a lot of money, so it was stipulated that only 2 jobs could be running at any time, and requests for new jobs when 2 are already running would be rejected (and the user notified that the server is busy).</p>

<p>My current solution uses an Executor from concurrent.futures, and requires setting the Apache server to run only one process, reducing responsiveness (current user count is very low, so it's okay for now).</p>

<p>If possible I would like to use Celery for this, but I did not see in the documentation any way to accomplish this particular setting.</p>

<p>How can I run up to a limited number of jobs in the background in a Django application, and notify users when jobs are rejected because the server is busy?</p>

<p>I have two solutions for this particular case, one an out of the box solution by celery, and another one that you implement yourself.</p>

<ol>
<li>You can do something like this with celery workers. In particular, you <strong>only create two worker processes with concurrency=1</strong> (or well, one with concurrency=2, but that's gonna be threads, not different processes), this way, only two jobs can be done asynchronously. Now you need a way to raise exceptions if both jobs are occupied, then you use <a href="http://docs.celeryproject.org/en/latest/userguide/monitoring.html#management-command-line-utilities-inspect-control" rel="nofollow">inspect</a>, to count the number of active tasks and throw exceptions if required. For implementation, you can checkout <a href="http://stackoverflow.com/questions/5544629/retrieve-list-of-tasks-in-a-queue-in-celery">this SO post</a>.</li>
</ol>

<p>You might also be interested in <a href="http://docs.celeryproject.org/en/latest/userguide/tasks.html#Task.rate_limit" rel="nofollow">rate limits</a>.</p>

<ol start="2">
<li><p>You can do it all yourself, using a locking solution of choice. In particular, a nice implementation that makes sure only two processes are running with redis (and redis-py) is as simple as the following. (Considering you know redis, since you know celery)</p>

<pre><code>from redis import StrictRedis

redis = StrictRedis('localhost', '6379')
locks = ['compute:lock1', 'compute:lock2']
for key in locks:
    lock = redis.lock(key, blocking_timeout=5)
    acquired = lock.acquire()
    if acquired:
        do_huge_computation()
        lock.release()
    else:
        raise SystemLimitsReached("Already at max capacity !")
</code></pre></li>
</ol>

<p>This way you make sure only two running processes can exist in the system. A third processes will block in the line <code>lock = redis.lock(key)</code> for <strong>blocking_timeout</strong> seconds, if the locking was successful, <code>acquired</code> would be True, else it's False and you'd tell your user to wait ! </p>

<p>I had the same requirement sometime in the past and what I ended up coding was something like the solution above. In particular</p>

<ol>
<li>This has the least amount of race conditions possible</li>
<li>It's easy to read</li>
<li>Doesn't depend on a sysadmin, suddenly doubling the concurrency of workers under load and blowing up the whole system.</li>
<li>You can also <strong><em>implement the limit per user</em></strong>, meaning each user can have 2 simultaneous running jobs, by only changing the lock keys from <em>compute:lock1</em> to <strong>compute:userId:lock1</strong> and lock2 accordingly. You can't do this one with vanila celery.</li>
</ol>

python django asynchronous celery background-process