airflow triggle_dag execution_date is the next day, why?
<p>Recently I have tested airflow so much that have one problem with <code>execution_date</code> when running <code>airflow trigger_dag &lt;my-dag&gt;</code>.</p>

<p>I have learned that <code>execution_date</code> is not what we think at first time from <a href="https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls" rel="nofollow">here</a>:</p>

<blockquote>
  <p>Airflow was developed as a solution for ETL needs. In the ETL world,
  you typically summarize data. So, if I want to summarize data for
  2016-02-19, I would do it at 2016-02-20 midnight GMT, which would be
  right after all data for 2016-02-19 becomes available.</p>
</blockquote>

<pre><code>start_date = datetime.combine(datetime.today(),
                              datetime.min.time())

args = {
    "owner": "xigua",
    "start_date": start_date
}
dag = DAG(dag_id="hadoopprojects", default_args=args,
          schedule_interval=timedelta(days=1))


wait_5m = ops.TimeDeltaSensor(task_id="wait_5m",
                              dag=dag,
                              delta=timedelta(minutes=5))
</code></pre>

<p>Above codes is the start part of my daily workflow, the first task is a TimeDeltaSensor that waits another 5 minutes before actual work, so this means my dag will be triggered at <code>2016-09-09T00:05:00</code>, <code>2016-09-10T00:05:00</code>... etc.</p>

<p>In Web UI, I can see something like <code>scheduled__2016-09-20T00:00:00</code>, and task is run at <code>2016-09-21T00:00:00</code>, which seems reasonable according to <code>ETL</code> model.</p>

<p>However someday my dag is not triggered for unknown reason, so I trigger it manually, if I trigger it at <code>2016-09-20T00:10:00</code>, then the TimeDeltaSensor will wait until <code>2016-09-21T00:15:00</code> before run. </p>

<p>This is not what I want, I want it to run at <code>2016-09-20T00:15:00</code> not the next day, I have tried passing <code>execution_date</code> through <code>--conf '{"execution_date": "2016-09-20"}'</code>, but it doesn't work.</p>

<p>How should I deal with this issue ?</p>

<pre><code>$ airflow version
[2016-09-21 17:26:33,654] {__init__.py:36} INFO - Using executor LocalExecutor
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
   v1.7.1.3
</code></pre>

<p>First, I recommend you use constants for <code>start_date</code>, because dynamic ones would act unpredictably based on with your airflow pipeline is evaluated by the scheduler.</p>

<p>More information about <code>start_date</code> here in an FAQ entry that I wrote and sort all this out:
<a href="http://pythonhosted.org/airflow/faq.html#what-s-the-deal-with-start-date" rel="nofollow">http://pythonhosted.org/airflow/faq.html#what-s-the-deal-with-start-date</a></p>

<p>Now, about <code>execution_date</code> and when it is triggered, this is a common gotcha for people onboarding on Airflow. Airflow sets <code>execution_date</code> based on the left bound of the schedule period it is covering, not based on when it fires (which would be the right bound of the period). When running an <code>schedule='@hourly'</code> task for instance, a task will fire every hour. The task that fires at 2pm will have an <code>execution_date</code> of 1pm because it assumes that you are processing the 1pm to 2pm time window at 2pm. Similarly, if you run a daily job, the run an with <code>execution_date</code> of <code>2016-01-01</code> would trigger soon after midnight on <code>2016-01-02</code>.</p>

<p>This left-bound labelling makes a lot of sense when thinking in terms of ETL and differential loads, but gets confusing when thinking in terms of a simple, cron-like scheduler.</p>

python airflow