Problem with NGINX, Kubernetes and CloudFlare

Question

I am experiencing exactly this issue: Nginx-ingress-controller fails to start after AKS upgrade to v1.22, with the exception that none of the proposed solutions is working for my case.

I am running a Kubernetes Cluster on Oracle Cloud and I accidentally upgraded the cluster and now I cannot connect anymore to the services through nginx-controller. After reading the official nginx documentation, I am aware of the new version of nginx, so I checked the documentation and re-installed the nginx-controller following Oracle Cloud official documentation.

I am able to perform step by step as I run: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44.0/deploy/static/provider/cloud/deploy.yaml

And then an ingress-nginx namespace is created and a LoadBalancer is created. Then as in the guide I have created a simple hello application (though not running on port 80):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: docker-hello-world
  labels:
    app: docker-hello-world
spec:
  selector:
    matchLabels:
      app: docker-hello-world
  replicas: 1
  template:
    metadata:
      labels:
        app: docker-hello-world
    spec:
      containers:
      - name: docker-hello-world
        image: scottsbaldwin/docker-hello-world:latest
        ports:
        - containerPort: 8088
---
apiVersion: v1
kind: Service
metadata:
  name: docker-hello-world-svc
spec:
  selector:
    app: docker-hello-world
  ports:
    - port: 8088
      targetPort: 8088
  type: ClusterIP

and then the ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world-ing
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - secretName: tls-secret
  rules:
  - http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: docker-hello-world-svc
              port:
                number: 8088

But when running the curl commands I only get a curl: (56) Recv failure: Connection reset by peer.

So I then tried to connect to some python microservices that are already running by simply editing the ingress, but whatever I do I get the same error message. And when setting the host as the following:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world-ing
  namespace: ingress-nginx
  annotations:
    kubernetes.io/ingress.class: "nginx"

spec:
  rules:
  - host: SUBDOMAIN.DOMAIN.com
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: ANY_MICROSERVICE_RUNNING_IN_CLUSTER
              port:
                number: EXPOSED_PORT_BY_MICROSERVICE

Then, by setting the subdomain on CloudFlare I only get a 520 Bad Gateway.

Can you help me find what is that I do not see?

Answer 1

This may be related to your Ingress resource.

In Kubernetes versions v1.19 and above, Ingress resources should use ingressClassName instead of the older annotation syntax. Additional information on what should be done when upgrading can be found on the official Kubernetes documentation.

However, with the changes it requires at face value, from the information you're provided so far, your Ingress resource should look this:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world-ing
spec:
  ingressClassName: nginx
  rules:
  - host: SUBDOMAIN.DOMAIN.com
    http:
      paths:
      - backend:
          service:
            name: docker-hello-world-svc
            port:
              number: 8088
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - SUBDOMAIN.DOMAIN.com
    secretName: tls-secret

Additionally, please provide the deployment Nginx-ingress logs if you still have issues, as the Cloudflare error does not detail what could be wrong apart from providing a starting point.

Your service definition is configured as a ClusterIP as well. It should be defined as LoadBalancer in order to receive external traffic. Otherwise, it'll have no external IP address.

As someone who uses Cloudflare and Nginx, there are multiple reasons why you're receiving a 520 error, so it'd be better if we could reduce the scope of what could be the main issue. Let me know if you have any questions.