introducing synchronous chained execution in celery workflow
<p>Suppose there are 4 tasks <code>T1</code>, <code>T2</code>, <code>T3</code>, <code>T4</code>. They are chained together as <code>T1.si() | T2.si() | T3.si() | T4.si()</code>. <code>T3</code> spawns further tasks <code>T30 .. T3n</code> asynchronously like <code>chord(T30,...,T3n)(reduce.s())</code>. I don't know <code>n</code> in advance (i.e., the no of subtasks <code>T3i</code>s that would be spawned).</p>

<p>I want <code>T4</code> to execute only after all of the <code>T3i..T3n</code> tasks complete. As expected, <code>T3</code> returns immediately due to the asynchronous behavior and then T4 starts executing before the chord is complete.</p>

<p>I could add a synchronous task that just does <code>T3.get()</code> before <code>T4</code>, but it will block one of the worker processes.</p>

<p>Is there a way to fix this design to avoid the blocking task, or a better design?</p>

<p>I will improve @jenner-felton's comment a bit...</p>

<p>You may call it like this:</p>

<pre><code>chain(T1.s(), T2.s(), T3.s(T4.s()))
</code></pre>

<p>e.g. <code>T4.s()</code> passed as one of parameters to the T3 task.</p>

<p>And T3 will run a <code>chord</code> itself with T4 passed as a callback.</p>

python asynchronous celery