GKE BackendConfig not working with customRequestHeaders

Question

I have a nodejs application running on Google Kubernetes Engine (v1.20.8-gke.900)

I want to add custom header to get client's Region and lat long so I refer to this article and this one also and created below kubernetes config file, but when I am printing the header I am not getting any custom header.

#k8s.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: my-app-ns-prod
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: npm-app-deployment
  namespace: my-app-ns-prod
  labels:
    app: npm-app-deployment
    tier: backend

spec:
  template:
    metadata:
      name: npm-app-pod
      namespace: my-app-ns-prod
      labels:
        app: npm-app-pod
        tier: backend

    spec:
      containers:
      - name: my-app-container
        image: us.gcr.io/img/my-app:latest
        ports:
        - containerPort: 3000
          protocol: TCP
        envFrom:
          - secretRef:
              name: npm-app-secret
          - configMapRef:
              name: npm-app-configmap
        imagePullPolicy: Always
      imagePullSecrets:
        - name: gcr-regcred

  replicas: 3
  minReadySeconds: 30

  selector:
    matchLabels:
      app: npm-app-pod
      tier: backend
---
apiVersion: v1
kind: Service
metadata:
  name: npm-app-service
  namespace: my-app-ns-prod
  annotations:
    cloud.google.com/backend-config: '{"ports": {"80":"npm-app-backendconfig"}}'
    cloud.google.com/neg: '{"ingress": true}'
spec:
  selector:
    app: npm-app-pod
    tier: backend
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 3000
    - name: https
      protocol: TCP
      port: 443
      targetPort: 3000
  type: LoadBalancer
---
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: npm-app-backendconfig
  namespace: my-app-ns-prod
spec:
  customRequestHeaders:
    headers:
    - "HTTP-X-Client-CityLatLong:{client_city_lat_long}"
    - "HTTP-X-Client-Region:{client_region}"
    - "HTTP-X-Client-Region-SubDivision:{client_region_subdivision}"
    - "HTTP-X-Client-City:{client_city}"
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /api/v1
            pathType: Prefix
            backend:
              service:
                name: npm-app-service
                port:
                  number: 80
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: npm-app-configmap
  namespace: my-app-ns-prod
data:
  APP_ID: "My App"
  PORT: "3000"
---
apiVersion: v1
kind: Secret
metadata:
  name: npm-app-secret
  namespace: my-app-ns-prod
type: Opaque
data:
  MONGO_CONNECTION_URI: ""
  SESSION_SECRET: ""

Answer 1

Actually the issue was with Ingress Controller, I missed to defined "cloud.google.com/backend-config". Once I had defined that I was able to get the custom header. Also I switched to GKE Ingress controller (gce) from nginx. But the same things works with nginx also

This is how my final ingress looks like.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    cloud.google.com/backend-config: '{"default": "npm-app-backendconfig"}'
    kubernetes.io/ingress.class: "gce"
spec:
  ...
  ...

Reference: User-defined request headers :