'How to customize error pages served via the default backend of an nginx ingress controller?

I'm running an Nginx Ingress Controller installed via Helm on my Kubernetes cluster. I would like to change the HTML/CSS in the default backend service for some specific errors (e.g. 404).

This link provides some general information about the default backend. However, there is no mention of how to actually customize the served HTML/CSS files.

nginx kubernetes

Solution 1:^[1]

August 2021 update:

The original answer contains the steps necessary to deploy a custom default backend on kubernetes.

However, the newest version of ingress-nginx allows the user to only specify the docker image to pull - no need for other k8s resource files (i.e. service and deployment).

My current values.yaml which I'm using for the nginx ingress controller to allow a custom default backend:

defaultBackend:
  enabled: true
  name: custom-default-backend
  image:
    repository: dvdblk/custom-default-backend
    tag: "latest"
    pullPolicy: Always
  port: 8080
  extraVolumeMounts:
    - name: tmp
      mountPath: /tmp
  extraVolumes:
    - name: tmp
      emptyDir: {}

Here is the GitHub repo of the custom backend.

Older answer:

Alright, some parts of these answers were helpful on the hunt for the complete solution, especially the one from @Matt. However, it took me quite some time to get this working so I've decided to write my own answer with all the necessary details that others might struggle with as well.

The first thing would be to create a Docker image server capable of responding to any request with 404 content, except /healthz and /metrics. As @Matt mentioned this could be an Nginx instance (which I've used). To sum up:

/healthz should return 200
/metrics is optional, but it should return data that is readable by Prometheus in case you are using it for k8s metrics. Nginx can provide some basic data that Prometheus can read. Consider this link if you would like to get the full Prometheus integration with Nginx.
/ returns a 404 with your custom HTML content.

Thus, the Dockerfile looks like this:

FROM nginx:alpine

# Remove default NGINX Config
# Take care of Nginx logging
RUN rm /etc/nginx/conf.d/default.conf && \
    ln -sf /dev/stdout /var/log/nginx/access.log && \
    ln -sf /dev/stderr /var/log/nginx/error.log

# NGINX Config
COPY ./default.conf /etc/nginx/conf.d/default.conf

# Resources
COPY content/ /var/www/html/

CMD ["nginx", "-g", "daemon off;"]

In the same folder where Dockerfile is located, create this default.conf Nginx configuration file:

server {
    root /var/www/html;
    index 404.html;

    location / {
        
    }

    location /healthz {
        access_log off;
        return 200 "healthy\n";
    }
    
    location /metrics {
        # This creates a readable and somewhat useful response for Prometheus
        stub_status on;
    }

    error_page 404 /404.html;
    location = /404.html {
        internal;
    }
}

At last, provide a content/404.html file with HTML/CSS to your own liking.

Now build the Docker image with:

docker build --no-cache -t custom-default-backend .

Tag this image so that it is ready to be pushed into DockerHub (or your own private docker registry):

docker tag custom-default-backend:latest <your_dockerhub_username>/custom-default-backend

Push the image to a DockerHub repository:

docker push <your_dockerhub_username>/custom-default-backend

Now comes the part of integrating this custom-default-backend image into the Helm installation. In order to do this, we first need to create this k8s resource file (custom_default_backend.yaml):

---
apiVersion: v1
kind: Service
metadata:
  name: custom-default-backend
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: custom-default-backend
    app.kubernetes.io/part-of: ingress-nginx
spec:
  selector:
    app.kubernetes.io/name: custom-default-backend
    app.kubernetes.io/part-of: ingress-nginx
  ports:
  - port: 80
    targetPort: 80
    name: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-default-backend
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: custom-default-backend
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: custom-default-backend
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: custom-default-backend
        app.kubernetes.io/part-of: ingress-nginx
    spec:
      containers:
      - name: custom-default-backend
        # Don't forget to edit the line below
        image: <your_dockerhub_username>/custom-default-backend:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 80

Assuming we have a k8s namespace ingress-nginx already created we can create these two resources.

kubectl apply -f custom_default_backend.yaml

Now in order to tie the Nginx Ingress Controller with our new service, we could probably just edit the deployment of the Ingress Controller. But I've decided to remove it completely via Helm:

helm delete nginx-ingress -n ingress-nginx

And install it again with this command (make sure you have the --set flag with proper arguments included):

helm install nginx-ingress --namespace ingress-nginx stable/nginx-ingress --set defaultBackend.enabled=false,controller.defaultBackendService=ingress-nginx/custom-default-backend

With these steps you should end up with a working custom default backend implementation. Here is a GitHub repo with the files that I have used in this answer.

Solution 2:^[2]

The project provides the Go custom error application that can be built into a container image to replace default-backend. The errorHandler function does the magic.

In the end it's a web server that responds to any request with 404 content, except /healthz and /metrics. You could do it with an nginx instance and html error pages if you want.

You probably don't want to use the full custom error handling, this is slightly different where the ingress controller will look for certain HTTP status codes from a regular app backend, and pass them to the default backend for handling. This causes issues for most application unless they were designed to use this from the outset.

Solution 3:^[3]

You have to create sample deployment with backend configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-errors
  labels:
    app.kubernetes.io/name: nginx-errors
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: nginx-errors
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: nginx-errors
        app.kubernetes.io/part-of: ingress-nginx
    spec:
      containers:
      - name: nginx-error-server
        image: quay.io/kubernetes-ingress-controller/custom-error-pages-amd64:0.3
        ports:
        - containerPort: 8080

Then you have to specify service for it:

apiVersion: v1
kind: Service
metadata:
  name: nginx-errors
  labels:
    app.kubernetes.io/name: nginx-errors
    app.kubernetes.io/part-of: ingress-nginx
spec:
  selector:
    app.kubernetes.io/name: nginx-errors
    app.kubernetes.io/part-of: ingress-nginx
  ports:
  - port: 80
    targetPort: 8080
    name: http

If you do not already have an instance of the NGINX Ingress controller running, deploy it according to the deployment guide, then follow these steps:

Edit the nginx-ingress-controller Deployment and set the value of the --default-backend-service flag to the name of the newly created error backend.
Edit the nginx-configuration ConfigMap and create the key custom-http-errors with a value of 404.

Pay attention on the IP address assigned to the NGINX Ingress controller Service.

$ kubectl get svc ingress-nginx
NAME            TYPE        CLUSTER-IP  EXTERNAL-IP   PORT(S)          AGE
ingress-nginx   ClusterIP   10.0.0.13   <none>        80/TCP,443/TCP   10m

The ingress-nginx Service is of type ClusterIP in this example. This may vary depending on your environment. Make sure you can use the Service to reach NGINX before proceeding with the rest of this example.

The NGINX configuration used for the backend pod is defined by the ConfigMaps. The binary ConfigMap creates the file returned to clients in the RPS tests. So if you want to make modification in following files specify them in configmap and apply changes.

Ingress Controller has one responsibility - implement Ingress rules defined in your cluster. To serve static content you should have a pod that does that, which indeed means ie. running second nginx in your stack. The thing is, that you should treat your ingress controller as part of the infrastructure providing generic cluster functionality, and serving static files from some place (or container if they are versioned/built as docker images) is de facto part of your application.

Take a look here: configmap-kubernetes-ingress, static-files-nginx-ingress default-backend, customization-ingress-errors.

Official documentation: nginx-ingress-controller.

Solution 4:^[4]

Use official ingress-nginx/nginx-errors image and map your custom pages from ConfigMap. This way you don't have to build own image.

See nginx-ingress official documentation https://kubernetes.github.io/ingress-nginx/examples/customization/custom-errors/

In case of deploying using Helm here is example values.yaml file.

# See https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml
controller:
    custom-http-errors: "404,500,503" # See https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#custom-http-errors
defaultBackend:
  enabled: true
  image:
    registry: k8s.gcr.io
    image: ingress-nginx/nginx-errors # Source https://github.com/kubernetes/ingress-nginx/tree/main/images/custom-error-pages
    tag: "0.48.1" # Check latest version on https://github.com/kubernetes/ingress-nginx/blob/main/docs/examples/customization/custom-errors/custom-default-backend.yaml
  extraVolumes:
  - name: error_page
    configMap:
      name: error_page
      items:
      - key: "error_page"
        path: "404.html"
      - key: "error_page"
        path: "500.html"
      - key: "error_page"
        path: "503.html"
  extraVolumeMounts:
  - name: error_page
    mountPath: /www

Example error page config map file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: error_page
  namespace: ingress-nginx
data:
  error_page: |
    <!DOCTYPE html>
    <html>
        <head>
            <title>ERROR PAGE</title>
        </head>
        <body>
          ERROR PAGE
        </body>
    </html>

Solution 5:^[5]

You need to create and deploy custom default backend which will return a custom error page.Follow the doc to deploy a custom default backend and configure nginx ingress controller by modifying the deployment yaml to use this custom default backend.

The deployment yaml for the custom default backend is here and the source code is here.

Solution 6:^[6]

There is another way to provide the custom error page in ingress-nginx, for that one has to modify the template of ingress-Nginx.(/etc/nginx/template).

     volumeMounts:
        - name: custom-errors
          mountPath: /usr/local/nginx/html/
          readOnly: true
        - name: nginx-ingress-template-volume
          mountPath: /etc/nginx/template
          readOnly: true

In the above YAML example, use path (/usr/local/nginx/html) for mounting the custom error pages. Inside the nginx template default server will look something like this (see below snippet).

# backend for when default-backend-service is not configured or it does not have endpoints
    server {
        listen {{ $all.ListenPorts.Default }} default_server {{ if $all.Cfg.ReusePort }}reuseport{{ end }} backlog={{ $all.BacklogSize }};
        {{ if $IsIPV6Enabled }}listen [::]:{{ $all.ListenPorts.Default }} default_server {{ if $all.Cfg.ReusePort }}reuseport{{ end }} backlog={{ $all.BacklogSize }};{{ end }}
        set $proxy_upstream_name "internal";

        access_log off;
        root /usr/local/nginx/html/;
        error_page 404 /404.html;
        error_page 500 502 503 504 /50x.html;
        location / {
          return 404;
        }
        location = /404.html {
          internal;
        }
        location = /50x.html {
          internal;
        }
    }

Make sure to provide the custom 404.html and 50xhtml pages inside the root (/usr/local/nginx/html/). Below snippet will help you mount the volume with custom pages.

   volumes:
    - name: custom-errors
      configMap:
        # Provide the name of the ConfigMap you want to mount.
        name: custom-ingress-pages
        items:
        - key: "404.html"
          path: "404.html"
        - key: "50x.html"
          path: "50x.html"
        - key: "index.html"
          path: "index.html"

This solution doesn't require you to spawn another service or pod of any kind to work this will be taken care of inside the ingress-nginx controller pod deployment(or daemonset). No need to warm up your cluster for extra service just for custom error messages (or pages).

For more reference: https://engineering.zenduty.com/blog/2022/03/02/customizing-error-pages

Solution 7:^[7]

The docker for default httpbackend can be found at : https://github.com/interlegis/nginx-ingress-controller-defaultbackend

Under root/etc/nginx folder, you can modify nginx.conf as per your convinience. Then build the update docker and deploy. ( you can check above answer how to deploy) Instead of removing ingress controller and then installing, its better to modify the deployed service so that it points to your pod, which you deployed to replace default http backend.

May be above answer can make thing easier for you. But if you are already operating production environment, I dont recommend to remove nginx ingress controller and install it back.

Solution 8:^[8]

I had this problem previously,.. but i got an idea to use route path "/" to a service.. so the ingress controller won't show the 404 page again.. And it works!!!

Create a deployment of simple service, in my case i'm using this with port 80
Then, expose the deployment as a service

kubectl expose deploy deployment-name --port=80
And the last step, we should configure ingress resource to route this service to "/" path


apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: default-ingress
  annotations:
    ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - http:
      paths:
        - path: /
          backend:
            serviceName: service-name
            servicePort: 80

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1
Solution 2	Matt
Solution 3	Malgorzata
Solution 4
Solution 5
Solution 6
Solution 7	NEERAJ SWARNKAR
Solution 8	Heri Fauzan

'How to customize error pages served via the default backend of an nginx ingress controller?

Solution 1:[1]

August 2021 update:

Older answer:

Solution 2:[2]

Solution 3:[3]

Solution 4:[4]

Solution 5:[5]

Solution 6:[6]

Solution 7:[7]

Solution 8:[8]