'Airflow - trigger a specific task from external
In Apache Airflow, let's say I want to set up a DAG that has 3 tasks.
Task A Task B Task C
When the DAG gets scheduled, I want task A to run, followed by Task B (when A completes). However, I want task C to only run when some external code triggers it (and it shouldn't poll and wait for an external condition to be satisfied; I want it to wait until it receives an external request to start execution, but only if A and B are completed). I also don't want to create another DAG for task C.
Is this possible? How to set up please? Will it require another task between B and C?
Thanks for any advice on how this can be achieved.
Solution 1:[1]
The best way to achieve this is to have two tasks leading into task C
, so for example:
A >> [B, x] >> C
Where x
is a new task. Then you can set x
to be your other trigger condition from somewhere external.
For example if you're waiting for a certain file to be delivered for C
to execute, create x
as a BranchOperator, and only return C
from it if the file exists.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|---|
Solution 1 | Thom Bedford |