Airflow task still running even after the endpoint process is completed

Question

I have an airflow task using SimpleHTTPOperator hitting a service in Google Cloud Run. I checked in Cloud Run Logs, the service itself is completed in 12 minutes but airflow task keep running up and i need to manually mark it to success.

Any idea what i should check? I'm new to airflow so i have no idea which logs i should check since there's no useful information from the task log.

I use LocalExecutor to run the DAGs.

Answer 1

In my team, we are using the HTTP Operator. However, we also created a custom operator based on it to call Google Cloud Run.

We have many dags that need to make multiple cloud run calls to do API calls for data extraction. Airflow calls Cloud Run and Cloud Run calls an API.

For some of these tasks, when the response time is higher than 8 minutes, Cloud Run sends a 200 back to Airflow, but the task keeps running indefinitely.

One suggestion we got from Google Cloud Support was using HttpSensor instead of HttpOperator. Aparrently, the operator run the tasks run asynchronously, but sensors can wait for some event and completion of tasks. You could try to use HttpSensor and see if it works.