'Why I can't see pause containers in GKE containerd CRI?

I am using a GKE cluster with Ubuntu nodes & containerd as CRI. I am not able to see any pause containers the way we used to see in docker. Are they no more exists in containerd? Anything changed or I am not aware of?

cluk_ti4@gke-cluster-1-default-pool-b897ab15-7kzn:~$ sudo -i

> root@gke-cluster-1-default-pool-b897ab15-7kzn:~# docker ps CONTAINER
> ID        IMAGE               COMMAND             CREATED            
> STATUS              PORTS               NAMES
> 
> root@gke-cluster-1-default-pool-b897ab15-7kzn:~# crictl ps CONTAINER  
> IMAGE               CREATED             STATE               NAME      
> ATTEMPT             POD ID 65b544036d815       cb2388d1f0a57       25
> hours ago        Running             data-db                0         
> 868cf5dd712a3 42afee3f328b1       0e403e3816e89       25 hours ago    
> Running             dbcontainer            0                  
> 1e67b12c7ddbf 944ac9c2334e2       295c7be079025       25 hours ago    
> Running             nginx                  0                  
> 9bc0d4292190b 99aff9af2f0c8       0e403e3816e89       25 hours ago    
> Running             redis                  0                  
> d5cda32e41f0f 43af76f1b819e       6266988902813       4 days ago      
> Running             prometheus-to-sd       0                  
> 43441f62220af 17d024b959956       d204263033d6e       4 days ago      
> Running             sidecar                0                  
> 43441f62220af e417d5e3b723f       ffd5a31c75009       4 days ago      
> Running             dnsmasq                0                  
> 43441f62220af d1c035046787f       1434d0253770f       4 days ago      
> Running             konnectivity-agent     0                  
> b19697ac36cf1 596e5a51c5fb8       1434d0253770f       4 days ago      
> Running             konnectivity-agent     0                  
> d0d35f65b0a3d 9231a60426be9       98b27a8d721c5       4 days ago      
> Running             gce-pd-driver          0                  
> e17e59a9486f6 63b943910b402       a26d732ed0895       4 days ago      
> Running             gke-metrics-agent      0                  
> 7052db1775ea5 cca9f35cec83d       5440bb4e13af5       4 days ago      
> Running             kubedns                0                  
> 43441f62220af 83570807e719c       ff9d4d52a7759       4 days ago      
> Running             fluentbit-gke          0                  
> 7bf90df2dc604 a0444e1f50435       8ee6ce05080ec       4 days ago      
> Running             csi-driver-registrar   0                  
> e17e59a9486f6 16ec942baf944       294aee909773c       4 days ago      
> Running             fluentbit              0                  
> 7bf90df2dc604 1cb29a3a26896       217b1e208caea       4 days ago      
> Running             kube-proxy             0                  
> 3923908ea54d7

root@gke-cluster-1-default-pool-b897ab15-7kzn:~# crictl ps | grep -i pause

root@gke-cluster-1-default-pool-b897ab15-7kzn:~#



Solution 1:[1]

The pause container still exists with containerd indeed however they're not visible to crictl. You should be able to see them though via a simple sudo ps command.

Solution 2:[2]

As LaudinCast wrote, effectively the pause containers still exist. It's needed for many things, such as holding the network namespace for the whole Pod. However, I'm still not sure why the CRI specification makes them filtered from the ListContainers method's result!

EDIT: Now I know, it's because the CRI has a concept of Sandbox and Containers in these sandboxes. See the RPC calls defined in the CRI spec proto. Effectively, crictl pods will list all the sandboxes, thus the pause containers in our situation. You can compare the ID, they will match, you can also use crictl -D pods to see the RPC in use, and compare with crictl -D ps.

For example, in the deprecated, and even removed in Kubernetes 1.24, Dockershim that implements the CRI for Docker, you can find the code line that will remove the pause containers from the list returned. It was apparently added back in 2016. Apparently, containerd does exactly the same in its CRI implementation, I did not take the time to find the exact position in the code base but it should be around here.

If you want to see the pause containers on a host running containerd, you can use ctr or nerdctl to list all the containers running. Just beware that containerd use namespaces for the containers (not just Linux namespaces lol). So you have to type something like nerdctl --namespace k8s.io ps or ctr --namespace=k8s.io c ls and everything will be listed just like with docker ps before!

If you need more details, see my "research" thread on the Kubernetes slack here.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 LundinCast
Solution 2