Reactor Netty Client ReadTimeoutException when using Http/2 causing PrematureCloseException

Question

I am using a reactor-netty client and setting up http/2 via alpn. When I setup a ReadTimeoutHandler via the .doOnConnected(...) hook, I end up getting this exception:

Jan 09, 2021 6:40:35 PM io.netty.channel.DefaultChannelPipeline onUnhandledInboundException
WARNING: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.timeout.ReadTimeoutException

This seems benign and the response comes fine, so I'm not really sure what's going on there, but when I start driving my client hard, particularly with gatling, I start to see failed responses via PrematureCloseException, which would not be acceptable in a production environment. This behavior goes away as soon as I remove the ReadTimeoutHandler.

I'm not sure what's going on here and I'm not sure if maybe I'm setting something up incorrectly due to Http/2. As far as I've read and as far as I can tell, this is the recommended way to extend the client and setup the ReadTimeoutHandler. This works perfectly over Http/1.1, I'm not sure what is changing over Http/2.

Here's a recreation of how the client can be setup to reproduce this -- simply use this client to hit your favorite http/2 endpoint.

I am using projectreactor bom version 2020.0.2

public static HttpClient createClient() {

final String[] alpnProtocols =
                new String[]{ApplicationProtocolNames.HTTP_2};
        // if the application protocol is set to null the netty code will set it to
        // JdkDefaultApplicationProtocolNegotiator.INSTANCE (which is package protected)
        final ApplicationProtocolConfig APPLICATION_PROTOCOL_CONFIG =
                new ApplicationProtocolConfig(
                        ApplicationProtocolConfig.Protocol.ALPN,
                        // NO_ADVERTISE is currently the only mode supported by both OpenSsl and JDK providers.
                        ApplicationProtocolConfig.SelectorFailureBehavior.NO_ADVERTISE,
                        // ACCEPT is currently the only mode supported by both OpenSsl and JDK providers.
                        ApplicationProtocolConfig.SelectedListenerFailureBehavior.ACCEPT,
                        alpnProtocols);
        final String[] SUPPORTED_PROTOCOLS =
                new String[]{"TLSv1.3"}; // note that http/2 only works with TLSv1.3 or TLSv1.2

        // a null for non http/2 will result in the default cipher suites for http1.1
        final List<String> CIPHER_SUITES =
                Arrays.asList(new String[]{ "TLS_AES_256_GCM_SHA384" });


        try {
            System.setProperty("io.netty.handler.ssl.noOpenSsl", "false"); // lazy way to set this up

            SslContext sslContext = SslContextBuilder
                    .forClient()
                    .sslProvider(io.netty.handler.ssl.SslProvider.OPENSSL)
                    .ciphers(CIPHER_SUITES)
                    .applicationProtocolConfig(APPLICATION_PROTOCOL_CONFIG)
                    .trustManager(InsecureTrustManagerFactory.INSTANCE)
                    .protocols(SUPPORTED_PROTOCOLS)
                    .build();


            return HttpClient.create()
                    .port(3443)
                    .baseUrl("https://localhost")
                    .secure(spec -> spec.sslContext(sslContext))
                    .protocol(HttpProtocol.H2)
                    .doOnConnected(con -> con.addHandlerLast(
                            new ReadTimeoutHandler(4_500, TimeUnit.MILLISECONDS)));


        } catch (Exception e) {
            throw new RuntimeException(e);
        }

Answer 1

PrematureCloseException is what you get when the connection get closed by the remote peer while Gatling is trying to write on it. It's a perfectly normal situation when reusing a pooled keep-alive connection as network is not instant.

Old versions of Gatling were not handling this case perfectly. If you're using an old version, you should upgrade (latest as of now is 3.5.0).