'Gunicorn worker terminated with signal 9

I am running a Flask application and hosting it on Kubernetes from a Docker container. Gunicorn is managing workers that reply to API requests.

The following warning message is a regular occurrence, and it seems like requests are being canceled for some reason. On Kubernetes, the pod is showing no odd behavior or restarts and stays within 80% of its memory and CPU limits.

[2021-03-31 16:30:31 +0200] [1] [WARNING] Worker with pid 26 was terminated due to signal 9

How can we find out why these workers are killed?

Solution 1:^[1]

I encountered the same warning message.

[WARNING] Worker with pid 71 was terminated due to signal 9

I came across this faq, which says that "A common cause of SIGKILL is when OOM killer terminates a process due to low memory condition."

I used dmesg realized that indeed it was killed because it was running out of memory.

Out of memory: Killed process 776660 (gunicorn)

Solution 2:^[2]

In our case application was taking around 5-7 minutes to load ML models and dictionaries into memory. So adding timeout period of 600 seconds solved the problem for us.

gunicorn main:app --workers 1 --worker-class uvicorn.workers.UvicornWorker --bind 0.0.0.0:8443 --timeout 600

Solution 3:^[3]

I was using AWS Beanstalk to deploy my flask application and I had a similar error.

In the log I saw:
web: MemoryError
[CRITICAL] WORKER TIMEOUT
[WARNING] Worker with pid XXXXX was terminated due to signal 9

I was using the t2.micro instance and when I changed it to t2.medium my app worked fine. In addition to this I changed to the timeout in my nginx config file.

Solution 4:^[4]

In my case the problem was in long application startup caused by ml model warm-up (over 3s)

Solution 5:^[5]

I encountered the same warning message when I limit the docker's memory, use like -m 3000m.

see docker-memory

and

gunicorn-Why are Workers Silently Killed?

The simple way to avoid this is set a high memory for docker or not set.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1	Simon
Solution 2	ACL
Solution 3	Vkey
Solution 4	EgurnovD
Solution 5	hstk

'Gunicorn worker terminated with signal 9

Solution 1:[1]

Solution 2:[2]

Solution 3:[3]

Solution 4:[4]

Solution 5:[5]