'Why I can't see pause containers in GKE containerd CRI?
I am using a GKE cluster with Ubuntu nodes & containerd as CRI. I am not able to see any pause containers the way we used to see in docker. Are they no more exists in containerd? Anything changed or I am not aware of?
cluk_ti4@gke-cluster-1-default-pool-b897ab15-7kzn:~$ sudo -i
> root@gke-cluster-1-default-pool-b897ab15-7kzn:~# docker ps CONTAINER
> ID IMAGE COMMAND CREATED
> STATUS PORTS NAMES
>
> root@gke-cluster-1-default-pool-b897ab15-7kzn:~# crictl ps CONTAINER
> IMAGE CREATED STATE NAME
> ATTEMPT POD ID 65b544036d815 cb2388d1f0a57 25
> hours ago Running data-db 0
> 868cf5dd712a3 42afee3f328b1 0e403e3816e89 25 hours ago
> Running dbcontainer 0
> 1e67b12c7ddbf 944ac9c2334e2 295c7be079025 25 hours ago
> Running nginx 0
> 9bc0d4292190b 99aff9af2f0c8 0e403e3816e89 25 hours ago
> Running redis 0
> d5cda32e41f0f 43af76f1b819e 6266988902813 4 days ago
> Running prometheus-to-sd 0
> 43441f62220af 17d024b959956 d204263033d6e 4 days ago
> Running sidecar 0
> 43441f62220af e417d5e3b723f ffd5a31c75009 4 days ago
> Running dnsmasq 0
> 43441f62220af d1c035046787f 1434d0253770f 4 days ago
> Running konnectivity-agent 0
> b19697ac36cf1 596e5a51c5fb8 1434d0253770f 4 days ago
> Running konnectivity-agent 0
> d0d35f65b0a3d 9231a60426be9 98b27a8d721c5 4 days ago
> Running gce-pd-driver 0
> e17e59a9486f6 63b943910b402 a26d732ed0895 4 days ago
> Running gke-metrics-agent 0
> 7052db1775ea5 cca9f35cec83d 5440bb4e13af5 4 days ago
> Running kubedns 0
> 43441f62220af 83570807e719c ff9d4d52a7759 4 days ago
> Running fluentbit-gke 0
> 7bf90df2dc604 a0444e1f50435 8ee6ce05080ec 4 days ago
> Running csi-driver-registrar 0
> e17e59a9486f6 16ec942baf944 294aee909773c 4 days ago
> Running fluentbit 0
> 7bf90df2dc604 1cb29a3a26896 217b1e208caea 4 days ago
> Running kube-proxy 0
> 3923908ea54d7
root@gke-cluster-1-default-pool-b897ab15-7kzn:~# crictl ps | grep -i pause
root@gke-cluster-1-default-pool-b897ab15-7kzn:~#
Solution 1:[1]
The pause container still exists with containerd indeed however they're not visible to crictl. You should be able to see them though via a simple sudo ps command.
Solution 2:[2]
As LaudinCast wrote, effectively the pause containers still exist. It's needed for many things, such as holding the network namespace for the whole Pod. However, I'm still not sure why the CRI specification makes them filtered from the ListContainers method's result!
EDIT: Now I know, it's because the CRI has a concept of Sandbox and Containers in these sandboxes. See the RPC calls defined in the CRI spec proto. Effectively, crictl pods will list all the sandboxes, thus the pause containers in our situation. You can compare the ID, they will match, you can also use crictl -D pods to see the RPC in use, and compare with crictl -D ps.
For example, in the deprecated, and even removed in Kubernetes 1.24, Dockershim that implements the CRI for Docker, you can find the code line that will remove the pause containers from the list returned. It was apparently added back in 2016. Apparently, containerd does exactly the same in its CRI implementation, I did not take the time to find the exact position in the code base but it should be around here.
If you want to see the pause containers on a host running containerd, you can use ctr or nerdctl to list all the containers running. Just beware that containerd use namespaces for the containers (not just Linux namespaces lol). So you have to type something like nerdctl --namespace k8s.io ps or ctr --namespace=k8s.io c ls and everything will be listed just like with docker ps before!
If you need more details, see my "research" thread on the Kubernetes slack here.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | LundinCast |
| Solution 2 |