Nginx Kubernetes Ingress returns 502

Question

I am trying to deploy simple REST API app in Kubernetes cluster in AWS EKS. Everything else is reverse proxying just fine.

I use following config for deployment. Pod is healthy, pod restarts every 20 minutes because AWS kills connection towards my development free tier db, but I run several replicas just in case.

When I try to connect to domain I get 502 Bad Gateway error. I really do not know what is going on.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: xyz-microservice
spec:
  replicas: 4
  template:
    nodeSelector:
        node.kubernetes.io/role: worker
    tolerations:
      - effect: NoSchedule
        key: node.kubernetes.io/role
        operator: Equal
        value: worker
  selector:
    matchLabels:
      app: xyz-microservice
  template:
    metadata:
      labels:
        app: xyz-microservice
    spec:
      hostname: xyz-microservice
      containers:
      - name: xyz-microservice
        image:  redacted
        imagePullPolicy: Always
        env:
        - name: DB_USERNAME
          value: redacted
        - name: DB_PASSWORD
          value: redacted
        - name: DB_HOST
          value: redacted
        - name: DB_PORT
          value: "5432"
        - name: DB_NAME
          value: redacted
        resources:
          requests:
            memory: "32Mi"
            cpu: "80m"
          limits:
            memory: "128Mi"
            cpu: "100m"
        ports:
        - containerPort: 8080

---
apiVersion: v1
kind: Service
metadata:
  name: xyz-microservice-service
spec:
  type: ClusterIP
  selector:
    app: xyz-microservice
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP

And this is ingress configuration:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "internal"
    forecastle.stakater.com/expose: "true"
    forecastle.stakater.com/appName: "XYZ Microservice"
    cert-manager.io/cluster-issuer: "letsencrypt-aws"
  name: xyz-microservice
spec:
  rules:
    - host: xyz.internal.staging.k8s.redacted
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: xyz-microservice-service
                port:
                  number: 8080
  tls:
    - hosts:
      - xyz.internal.staging.k8s.redacted
      secretName: xyz-microservice-tls

It would mean alot if someone could guide me through.

Answer 1

Just keep in mind that almost every time you got a 502 bad gateway in nginx-ingress, it's linked to a backend port binding problem / bad backend configuration names etc.

Here your nginx ingress is configured to redirect the traffic from "xyz.internal.staging.k8s.redacted" host to the port 8080 of your "xyz-microservice-service" kubernetes service.

However 8080 is the targetPort of your service. No the exposed port that is in fact 80.

So, to fix this issue just change the port of the backend service, like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "internal"
    forecastle.stakater.com/expose: "true"
    forecastle.stakater.com/appName: "XYZ Microservice"
    cert-manager.io/cluster-issuer: "letsencrypt-aws"
  name: xyz-microservice
spec:
  rules:
    - host: xyz.internal.staging.k8s.redacted
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: xyz-microservice-service
                port:
                  number: 80
  tls:
    - hosts:
      - xyz.internal.staging.k8s.redacted
      secretName: xyz-microservice-tls

(Or reconfigure the port of your service to match the port of the ingress configuration.)