Why the k8s rollback (rollout undo) is not working?

Question

After a successful

kubectl rollout restart deployment/foo

the

kubectl rollout undo deployment/foo

or

kubectl rollout undo deployment/foo --to-revision=x

are not having effect. I mean, the pods are replaced by new ones and a new revision is created which can be checked with

kubectl rollout history deployment foo

but when I call the service, the rollback had no effect.

I also tried to remove the imagePullPolicy: Always, guessing that it was always pulling even in the rollback, with no success because probably one thing is not related to the other.

---

Edited: The test is simple, I change the health check route of the http api to return something different in the json, and it doesn't.

---

Edited:

Maybe a typo, but not: I was executing with ... undo deployment/foo ..., and now tried with ... undo deployment foo .... It also gives me deployment.apps/foo rolled back, but no changes in the live system.

More tests: I changed again my api route to test what would happen if I executed a rollout undo to every previous revision one by one. I applied the last 10 revisions, and nothing.

Answer 1

To be able to rollback to a previous version don't forget to append the --record parameter to your kubectl command, for example:

kubectl apply -f DEPLOYMENT.yaml --record

Then you should be able to see the history as you know with:

kubectl rollout history deployment DEPLOYMENT_NAME

And your rollback will work properly

kubectl rollout undo deployment DEPLOYMENT_NAME --to-revision=CHOOSEN_REVISION_NUMBER

Little example:

consider my nginx deployment manifest "nginx-test.yaml" here:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

lets create it:

? kubectl apply -f nginx-test.yaml --record
Flag --record has been deprecated, --record will be removed in the future
deployment.apps/nginx-deployment created

lets check the image of this deployment, as expected from the manifest:

? k get pod nginx-deployment-74d589986c-k9whj -o yaml | grep image:
  - image: nginx
    image: docker.io/library/nginx:latest

now lets modify the image of this deployment to "nginx:1.21":

#"nginx=" correspond to the name of the container inside the pod create by the deployment.
? kubectl set image deploy nginx-deployment nginx=nginx:1.21.6
deployment.apps/nginx-deployment image updated

we can optionnaly check the rollout status:

? kubectl rollout status deployment nginx-deployment
deployment "nginx-deployment" successfully rolled out

we can check the rollout history with:

? kubectl rollout history deploy nginx-deployment
deployment.apps/nginx-deployment
REVISION  CHANGE-CAUSE
1         kubectl apply --filename=nginx-test.yaml --record=true
2         kubectl apply --filename=nginx-test.yaml --record=true

lets check the image of this deployment, as expected:

? k get pod nginx-deployment-66dcfc79b5-4pk7w -o yaml | grep image:
  - image: nginx:1.21.6
    image: docker.io/library/nginx:1.21.6

Oh, no, i don't like this image ! Lets rollback:

? kubectl rollout undo deployment nginx-deployment --to-revision=1
deployment.apps/nginx-deployment rolled back

creating:

> kubectl get pod -o wide
NAME                                    READY   STATUS              RESTARTS   AGE     IP           NODE                   NOMINATED NODE   READINESS GATES
pod/nginx-deployment-66dcfc79b5-4pk7w   1/1     Running             0          3m41s   10.244.3.4   so-cluster-1-worker3   <none>           <none>
pod/nginx-deployment-74d589986c-m2htr   0/1     ContainerCreating   0          13s     <none>       so-cluster-1-worker2   <none>           <none>

after few seconds:

NAME                                    READY   STATUS    RESTARTS   AGE   IP            NODE                   NOMINATED NODE   READINESS GATES
pod/nginx-deployment-74d589986c-m2htr   1/1     Running   0          23s   10.244.4.10   so-cluster-1-worker2   <none>           <none>

as you can see it worked:

? k get pod nginx-deployment-74d589986c-m2htr -o yaml | grep image:
  - image: nginx
    image: docker.io/library/nginx:latest

lets recheck the history:

? kubectl rollout history deploy nginx-deployment
deployment.apps/nginx-deployment
REVISION  CHANGE-CAUSE
1         kubectl apply --filename=nginx-test.yaml --record=true
2         kubectl apply --filename=nginx-test.yaml --record=true

you can change the rollout history's CHANGE-CAUSE with the "kubernetes.io/change-cause" annotation:

? kubectl annotate deploy nginx-deployment kubernetes.io/change-cause="update image from 1.21.6 to latest" --reco
rd
Flag --record has been deprecated, --record will be removed in the future
deployment.apps/nginx-deployment annotated

lets recheck the history:

? kubectl rollout history deploy nginx-deployment
deployment.apps/nginx-deployment
REVISION  CHANGE-CAUSE
2         kubectl apply --filename=nginx-test.yaml --record=true
3         update image from 1.21.6 to latest

lets describe the deployment:

? kubectl describe deploy nginx-deploy
Name:                   nginx-deployment
Namespace:              so-tests
CreationTimestamp:      Fri, 06 May 2022 00:56:09 -0300
Labels:                 app=nginx
Annotations:            deployment.kubernetes.io/revision: 3
                        kubernetes.io/change-cause: update image from latest to latest
...

hope this has helped you, bguess.