Dynamically created tasks/dags are not working in apache airflow
<p>I'm created a DAG with scheduling @daily interval and separate task id for every workflow. But Its is not running as excepted. Is it possible to do like this? Is there any other ways to create dynamic tasks for a particular dag? And to pause a particular task instance using command line?</p>

<pre><code>from __future__ import print_function
from builtins import range
from airflow.operators import PythonOperator,DummyOperator,BranchPythonOperator,SqlSensor
from airflow.models import DAG
from datetime import datetime, timedelta

import time
from pprint import pprint

seven_days_ago = datetime.combine(
        datetime.today() - timedelta(7), datetime.min.time())

args = {
    'owner': 'varakumar',
    'start_date': seven_days_ago,
}

dag = DAG(
    dag_id='dynamic_task_creation', default_args=args,
    schedule_interval="@daily")

def get_decision():
    return "right"

start = DummyOperator(
    task_id='start',
    dag=dag)

td=datetime.today()
x=str(datetime(td.year,td.month,td.day,td.hour,td.minute,td.second)).replace (" ", "_").replace (":", "-")
pause_task_id = ("pause-%s" % x)

pause = DummyOperator(
    task_id=pause_task_id,
    dag=dag)
pause.set_upstream(start)

decision = BranchPythonOperator(
    task_id='decision',
    python_callable=lambda: get_decision(),
    dag=dag)
decision.set_upstream(pause)

left = DummyOperator(
    task_id='left',
    dag=dag)
left.set_upstream(decision)

right = DummyOperator(
    task_id='right',
    dag=dag)
right.set_upstream(decision)
</code></pre>

<p>Thank you in advance</p>

<p>The first problem I see is that you're using a dynamic <code>start_date</code>.  I've seen some weird behavior when doing that, and I think it's based on the way that airflow maintains its list of past dagruns.  Try to specify a fixed <code>start_date</code> and see if that resolves anything.</p>

<p>In either case, the airflow docs <a href="https://airflow.incubator.apache.org/code.html#airflow.models.BaseOperator" rel="nofollow">advise against dynamic start dates</a> (scroll down a bit and read the description for <code>start_date</code>).</p>

<p>EDIT:  Also check <a href="https://airflow.incubator.apache.org/faq.html#what-s-the-deal-with-start-date" rel="nofollow">this</a> out for more info on that.</p>

python airflow