'How to push xcom from AwsGlueJobOperator when this task fails

I am trying to get xcom for a glue job run to get it's glueid. I need this to display the cloudwatch link on airflow output console in case the glue job fails. I am using AwsGlueJobOperator for running this glue job. But when the task fails it returns "none" as xcom value. And when it is successful , i get the glue_id. So, how to get the glue_id from this operator when this task fails? My task is:

trigger_glue_job = AwsGlueJobOperator(
        task_id='trigger_glue_job',
        job_name='test_glue',
        job_desc='load test data source',
        script_args=script_args,
        num_of_dpus=2,
        run_job_kwargs=json.loads(Variable.get("hourly_run_job_kwargs"))

    )

The operator in which i am calling that xcom:


    cloud_watch_output = BashOperator(
            task_id='cloud_watch_output',
            bash_command="echo 'For more details, Please check the Glue Logs here link_of_the_cloudwatch{{ ti.xcom_pull(task_ids=\'trigger_glue_job\') }}'",
            trigger_rule="all_done"
          )

airflow

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source

'How to push xcom from AwsGlueJobOperator when this task fails

Sources

Related Questions