'Kubernetes ingress controller - Error: ImagePullBackOff

I'm unable to get the controller working. Tried many times and still I get Error: ImagePullBackOff.

Is there a alternative that I can try or any idea why its failing?

    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.27.0/deploy/static/mandatory.yaml

kubectl describe pod nginx-ingress-controller-7fcb6cffc5-m8m5c -n ingress-nginx

Name:         nginx-ingress-controller-7fcb6cffc5-m8m5c
Namespace:    ingress-nginx
Priority:     0
Node:         ip-10-0-0-244.ap-south-1.compute.internal/10.0.0.244
Start Time:   Mon, 07 Dec 2020 08:21:13 -0500
Labels:       app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              pod-template-hash=7fcb6cffc5
Annotations:  kubernetes.io/limit-ranger: LimitRanger plugin set: cpu, memory request for container nginx-ingress-controller
              kubernetes.io/psp: eks.privileged
              prometheus.io/port: 10254
              prometheus.io/scrape: true
Status:       Pending
IP:           10.0.0.231
IPs:
  IP:           10.0.0.231
Controlled By:  ReplicaSet/nginx-ingress-controller-7fcb6cffc5
Containers:
  nginx-ingress-controller:
    Container ID:
    Image:         quay.io/kubernetes-ingress-controller/nginx-ingress-controller:master
    Image ID:
    Ports:         80/TCP, 443/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --configmap=$(POD_NAMESPACE)/nginx-configuration
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      --publish-service=$(POD_NAMESPACE)/ingress-nginx
      --annotations-prefix=nginx.ingress.kubernetes.io
    State:          Waiting
      Reason:       ErrImagePull
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=10s period=10s #success=1 #failure=3
    Readiness:  http-get http://:10254/healthz delay=0s timeout=10s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       nginx-ingress-controller-7fcb6cffc5-m8m5c (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from nginx-ingress-serviceaccount-token-xtnz9 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  nginx-ingress-serviceaccount-token-xtnz9:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nginx-ingress-serviceaccount-token-xtnz9
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/os=linux
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  19s   default-scheduler  Successfully assigned ingress-nginx/nginx-ingress-controller-7fcb6cffc5-m8m5c to ip-10-0-0-244.ap-south-1.compute.internal
  Normal   Pulling    18s   kubelet            Pulling image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:master"
  Warning  Failed     3s    kubelet            Failed to pull image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:master": rpc error: code = Unknown desc = Error response from daemon: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning  Failed     3s    kubelet            Error: ErrImagePull
  Normal   BackOff    3s    kubelet            Back-off pulling image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:master"
  Warning  Failed     3s    kubelet            Error: ImagePullBackOff

kubernetes

Solution 1:^[1]

It's failing because kubernetes cannot download the specified image. Check the events section Warning Failed 3s kubelet Failed to pull image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:master": rpc error: code = Unknown desc = Error response from daemon: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Maybe you dont have internet connectivity or this image does not exist. You can try running docker pull quay.io/kubernetes-ingress-controller/nginx-ingress-controller:master from your computer

Solution 2:^[2]

I had the same problem, with the ingress-nginx installation.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.2.0/deploy/static/provider/cloud/deploy.yaml

For some reason it couldn't get the ingress-nginx-controller.

$ kubectl get pods --namespace=ingress-nginx
NAME                                        READY   STATUS             RE
ingress-nginx-admission-create-6q4wx        0/1     Completed          0 
ingress-nginx-admission-patch-fr5ct         0/1     Completed          1 
ingress-nginx-controller-686556747b-dg68h   0/1     ImagePullBackOff   0

What I did was, I ran $ kubectl describe pod ingress-nginx-controller-686556747b-dg68h --namespace ingress-nginx and got the following output:

Name:         ingress-nginx-controller-686556747b-dg68h
Namespace:    ingress-nginx
Priority:     0
Node:         docker-desktop/x.x.x.x
Start Time:   Wed, 11 May 2022 20:11:55 +0430
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/name=ingress-nginx
              pod-template-hash=686556747b
Annotations:  <none>
Status:       Pending
IP:           x.x.x.x
IPs:
  IP:           x.x.x.x
Controlled By:  ReplicaSet/ingress-nginx-controller-686556747b
Containers:
  controller:
    Container ID:
    Image:         k8s.gcr.io/ingress-nginx/controller:v1.2.0@sha256:d819
    Image ID:
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-controller-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s perio
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s perio
    Environment:
      POD_NAME:       ingress-nginx-controller-686556747b-dg68h (v1:metad
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)

From Containers.controller.Image, I got the image name that kubernetes is trying to download but is unsuccessful to do so and tried to docker pull that image myself like so:

docker pull k8s.gcr.io/ingress-nginx/controller:v1.2.0@sha256:d819

Docker could pull the image successfully and after that everything worked just fine.

Solution 3:^[3]

As mentioned by John, creating a nat router and nat config allowed docker images to be pulled when I was facing the same issue. If you create a vpc native GKE cluster which is private it by default has no access to the internet. Unless you deploy a NAT router.

gcloud compute routers create nat-router \
    --network my-vpc \
    --region us-east4

gcloud compute routers nats create nat-config \
    --router-region us-east4 \
    --router nat-router \
    --nat-all-subnet-ip-ranges \
    --auto-allocate-nat-external-ips

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1	Kroustou
Solution 2	Cid
Solution 3	Chris Tatro

'Kubernetes ingress controller - Error: ImagePullBackOff

Solution 1:[1]

Solution 2:[2]

Solution 3:[3]

Sources

Related Questions

Solution 1:^[1]

Solution 2:^[2]

Solution 3:^[3]