Unable to connect to gRPC server in Kubernetes cluster, but can connect when I port-forward [Connection refused]

Question

Been stuck on this error for the past few days!

I have an HTTP server that is meant to connect with the gRPC server through the client. It works fine on my local machine when I start the gRPC server and start my HTTP server. However, When I try to deploy it in a cluster, the HTTP server is unable to connect with the error message error receiving stream from timer rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.109.237.114:5996: connect: connection refused"

What I find particularly weird is that if I forward the gRPC server from the cluster, my local HTTP server connects to it just fine. Inspecting the connection within the cluster I can see that the port is open but it still refuses connections. image of netstat inspection

Notes

• Experiene this issue on minikube and DOKS.
• Built these images on an M1 mac.
• There is no gRPC authenticationg => rpc.DialContext(ctx, serverAddress, grpc.WithTransportCredentials(insecure.NewCredentials()))

GRPC SERVER FILE

apiVersion: apps/v1
kind: Deployment
metadata:
  name: x-service
  labels:
    type: xx
    service: x-svc
spec:
  replicas: 1
  selector:
    matchLabels:
      type: xx
      service: x-svc
  template:
    metadata:
      labels:
        type:xx
        service: x-svc
    spec:
      containers:
        - name: x-api
          image: x/image

---

apiVersion: v1
kind: Service
metadata:
  name: x-service
spec:
  ports:
    - protocol: TCP
      port: 5996
      targetPort: 5996
  selector:
    type: xx
    service: x-svc

HTTP SERVER FILE

apiVersion: apps/v1
kind: Deployment
metadata:
  name: b-service
  labels:
    type: be
    service: be-svc
spec:
  replicas: 1
  selector:
    matchLabels:
      type: be
      service: be-svc
  template:
    metadata:
      labels:
        type: be
        service: be-svc
    spec:
      containers:
        - name: bapi
          image: x/grpc
          imagePullPolicy: Always
          env:
            - name: X_ADDRESS
              value: x-service:5996

---
apiVersion: v1
kind: Service
metadata:
  name: b-api-svc
spec:
  type: NodePort
  ports:
    - port: 8080
  selector:
    type: be
    service: be-svc

Answer 1

A few days ago I was running into a similar situation as you are: "Locally you can communicate with the gRPC service via port-forward, but it fail to communicate inside the Cluster (between Pods)"

If both the Pods are inside the same cluster, you should use the service name and port-number instead of the host name. In your case this should be x-service:5996. If you have a namespace then it should be x-service.<enter-namespace-here>:5996

In the Java context, your code should look similar to this:

ManagedChannel channel = ManagedChannelBuilder
  .forTarget("x-service:5996")
  .usePlaintext()
  .build();

If I am correct, within the k8s cluster your pods can interact with each other using their services names (and ports). However, if you want to establish communication via Ingress, check this documentation https://doc.traefik.io/traefik/routing/providers/kubernetes-ingress/

Hopefully this helps you.