Debug Alpine Image in K8s: No `netstat`, no `ip`, no `apk`

Question

There is a container in my Kubernetes cluster which I want to debug.

But there is nonetstat, no ip and no apk.

Is there a way to upgrade this image, so that the common tools are installed?

In this case it is the nginx container image in a K8s 1.23 cluster.

Answer 1

Alpine is a stripped-down version of the image to reduce the footprint. So the absence of those tools is expected. Although since Kubernetes 1.23, you can use the kubectl debug command to attach a debug pod to the subject pod. Syntax:

kubectl debug -it <POD_TO_DEBUG> --image=ubuntu --target=<CONTAINER_TO_DEBUG> --share-processes

Example: In the below example, the ubuntu container is attached to the Nginx-alpine pod, requiring debugging. Also, note that the ps -eaf output shows nginx process running and the cat /etc/os-release shows ubuntu running. The indicating process is shared/visible between the two containers.

ps@kube-master:~$ kubectl debug -it nginx --image=ubuntu --target=nginx --share-processes
Targeting container "nginx". If you don't see processes from this container, the container runtime doesn't support this feature.
Defaulting debug container name to debugger-2pgtt.
If you don't see a command prompt, try pressing enter.
root@nginx:/# ps -eaf
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 19:50 ?        00:00:00 nginx: master process nginx -g daemon off;
101           33       1  0 19:50 ?        00:00:00 nginx: worker process
101           34       1  0 19:50 ?        00:00:00 nginx: worker process
101           35       1  0 19:50 ?        00:00:00 nginx: worker process
101           36       1  0 19:50 ?        00:00:00 nginx: worker process
root         248       0  1 20:00 pts/0    00:00:00 bash
root         258     248  0 20:00 pts/0    00:00:00 ps -eaf
root@nginx:/# 

Debugging as ubuntu as seen here, this arm us with all sort of tools:

root@nginx:/# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
root@nginx:/# 

In case ephemeral containers need to be enabled in your cluster, then you can enable it via feature gates as described here.

Answer 2

The whole point of using containers is to optimize the resource utilization in your cluster. The images used should only include the packages that are needed to run your app.

The unwanted packages should be removed from your images (especially in prod) to reduce the compute utilization and to reduce the attack vector.

This appears to be a stripped down image that has only the libraries needed to run that application.

In order to debug, you will have to create a new container in the same pid and network namespace as the container you are trying to debug

Build container first

Dockerfile

FROM alpine
RUN apk update && apk add strace
CMD ["strace", "-p", "1"]

Build

$ docker build -t strace .

Run

docker run -t --pid=container:<targetContainer> \
  --net=container:targetContainer \
  --cap-add sys_admin \
  --cap-add sys_ptrace \
  strace
strace: Process 1 attached
futex(0xd72e90, FUTEX_WAIT, 0, NULL

https://rothgar.medium.com/how-to-debug-a-running-docker-container-from-a-separate-container-983f11740dc6