keep grpc channel alive without activity

Question

We have a system where clients open bi-directional grpc stream to ALB, which proxies to one of active server. So

              bi-di                      client <----------->  ALB  <--------> server

In-case of any failure of connection, clients re-connects to us as we want to keep a bi-di channel open & active. 

Question is : How can we keep the channel alive even if there is no activity for sometime. ALB are configured with 300 sec idle-timeout which means it will drop the connection if no packets are exchanged in 300 sec. 

I read on grpc page at https://grpc.io/blog/grpc-on-http2/#keeping-connections-alive , we should use keep alive settings on both sides. So I tried below configuration

bi-di client channel with :  keepAliveWithoutCalls(true).keepAliveTime(90, TimeUnit.SECONDS).keepAliveTimeout(10, TimeUnit.SECONDS)

And

Server is configured with : permitKeepAliveWithoutCalls(true).permitKeepAliveTime(1, TimeUnit.MINUTES)

But I received INTERNAL: HTTP/2 error code: PROTOCOL_ERROR Received Rst Stream after exactly 5 minutes. Which looks like ALB has dropped the connection after 5 minutes.

Any idea how we can keep idle connection alive ?

Answer 1

We can't use raw http2 pings as ALB doesn't support it HTTP2 PING frames over AWS ALB (gRPC keepalive ping).

I fixed above with small implementation both at client and server side :

 a) Client sends some dummy request to server every 1 min. This is a actual request defined in proto buf by passing some type like dummy request.

   b) On reception of every such above request, server responds back with a dummy response, which client ignores based on request type like dummy response.

This way cycle is whole complete and LB thinks that there is some activity on the http connection and it doesn't drop the connection.