Testing taskqueues locally on google app engine using dev_appserver.py
<p>My app engine app has 2 queues. A worker from the first queue initiates several workers in the second queue and wait for them to complete and consolidate.</p>

<p>This works fine when I deploy it, but it <s>doesn't work</s> never spawns the tasks in the second queue when testing locally. It gets stuck while waiting for it to update (see code below). The reason seems to be that dev_appserver is single threaded and does not launch services in the background.</p>

<p>This is really slowing down my development since I have to deploy to the cloud to test anything.</p>

<p>Any ways around this?</p>

<p>Edit: Adding the (sort of) pseudo code below. Again, this works perfectly fine in the cloud because I have a 10 minute time limit to complete the task and each <code>BigTask</code> takes not more than 30-40 seconds to complete.</p>

<h3>Queue 1 Worker</h3>

<pre><code>class BigTask(webapp2.RequestHandler):
  def do_stuff(self):
    #do something here
    small_task1 = taskqueue.add(...)
    small_task2 = taskqueue.add(...)
    small_task3 = taskqueue.add(...)
    small_task4 = taskqueue.add(...)
    # Create ndb entries for all small tasks

    while True:
      time.sleep(2)
      # Check if ndb entry updated for all small_tasks
      if status == 'Completed': #for all
        break

    add_results(...) # of all tasks here

    # Update ndb entry for big_task
    # done
</code></pre>

<h3>Queue 2 Worker</h3>

<pre><code>class SmallTask(webapp2.RequestHandler): 
  def do_stuff(self):
    # Do processing
    # Update ndb entry
    # done
</code></pre>

<p>Generally sleeping in GAE app code is not a good idea - wasting instance uptime, increasing request latency, risk of exceeding the request deadline etc.</p>

<p>You can achieve the same functionality by enqueuing a delayed BigTask (using the <code>countdown</code> option) which would check the status for all smaller tasks, do the <code>add_results(...)</code> stuff if they completed or, if not, re-queue itself (again delayed) to re-check later on.</p>

<p>As a side effect you could even use the same task queue for both kind of tasks (if that's inline with your app requirements, of course).</p>

python google-app-engine task-queue