Nginx ingress controller not giving metrics for prometheus

Question

I am trying to deploy an nginx ingress controller which can be monitored using prometheus however I am running into an issue that it seems no metrics pod(s) is being created like most posts and docs I have found online show.

I'm using helm to deploy the ingress controller and using a CLI arguement to enable metrics.

helm install ingress stable/nginx-ingress --set controller.metrics.enabled=true

Here is my ingress file

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    # add an annotation indicating the issuer to use.
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-dev"
    # needed to allow the front end to talk to the back end
    nginx.ingress.kubernetes.io/cors-allow-origin: "https://app.domain.com"
    nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, PATCH, OPTIONS"
    # needed for monitoring
    prometheus.io/scrape: "true"
    prometheus.io/port: "10254"
  name: dev-ingress
  namespace: development
spec:
  rules:
  - host: api.<domain>.com
    http:
      paths:
      - backend:
          serviceName: api
          servicePort: 8090
        path: /
  tls: # < placing a host in the TLS config will indicate a certificate should be created
  - hosts:
    - api.<domai>.com
    secretName: dev-ingress-cert # < cert-manager will store the created certificate in this secre

In case this makes a difference I am using the prometheus operator helm chart with the below command.

helm install monitoring stable/prometheus-operator --namespace=monitoring

All namespaces exist already so that shouldn't be an issue, as for the development vs monitoring name spaces I saw in many places this was acceptable so I went with it to make things easier to figure out what is happening.

Answer 1

I would follow this guide to setup monitoring for Nginx ingress controller. I believe what you are missing is a prometheus.yaml which defines scrape config for the Nginx ingress controller and RBAC for prometheus to be able to scrape the Nginx ingress controller metrics.

Edit: Annotate nginx ingress controller pods

kubectl annotate pods nginx-ingress-controller-pod prometheus.io/scrape=true -n ingress-nginx --overwrite

kubectl annotate pods nginx-ingress-controller-pod prometheus.io/port=10254  -n ingress-nginx --overwrite

Answer 2

I was not using helm but manifests and the method POD annotations to install the Prometheus. I follow the official doc but could not see the metric either.

I believe the Deployment manifest has some issues. The annotation shouldn't be put on the Deployment level but on the pod level

apiVersion: v1
kind: Deployment
..
spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "10254"
    label:
      ...
  ports:
    - name: prometheus
      containerPort: 10254
      ..

Also, I've confirmed the metric for Nginx is enabled by default when using manifest deployment. No extra steps are needed for this.