'Is it possible to use an XCom from an operator parameter without using Jinja?
Is it possible to pass an XCom to an operator parameter without using a Jinja template? I have a dict stored in an XCom and I need to pass it to an Operator that receives a dict, and using Jinja I can only return the XCom as string. i.e:
def push_xcom(ti):
ti.xcom_push(key='my_xcom', value={"key": "value"})
with DAG(...) as dag:
start = PythonOperator(task_id='dp_start', python_callable=push_xcom)
middle = MyOperator(task_id='dp_middle',
parameter='{{ti.xcom_pull(key="my_xcom")}}'
)
start >> middle
I know is possible using task.output as a parameter and store the Xcom using return in the python callable, but this create a dependency between tasks and I need to use the XCom in another task... For example:
def push_xcom(ti):
return {"key": "value"}
def pull_xcom(input):
print(f'XCom: {input}')
with DAG(...) as dag:
start = PythonOperator(task_id='dp_start',
python_callable=push_xcom)
middle = MyOperator(task_id='dp_middle',
parameter=start.output)
end = PythonOperator(task_id='dp_start', python_callable=pull_xcom,
op_kwargs={'input': start.output})
start >> middle >> end
So is it possible to do it without using Jinja Templates or task.output? Or by passing an XCom to more than one task will I irremediably create dependencies between them (visible from the Graph view)?
Thanks in advance.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|