grpc and polly - .net core 6

Question

I'm trying to use Polly as retry policy handler for grpc in my .net core 6 project. I noticed that the retryFunc is never invoked. I started from this project gRPC & ASP.NET Core 3.1: Resiliency with Polly

class Program
{
    static async Task Main(string[] args)
    {
        // DI
        var services = new ServiceCollection();

        var loggerFactory = LoggerFactory.Create(logging =>
        {
            logging.AddConsole();
            logging.SetMinimumLevel(LogLevel.Debug);
        });

        var serverErrors = new HttpStatusCode[] { 
            HttpStatusCode.BadGateway, 
            HttpStatusCode.GatewayTimeout, 
            HttpStatusCode.ServiceUnavailable, 
            HttpStatusCode.InternalServerError, 
            HttpStatusCode.TooManyRequests, 
            HttpStatusCode.RequestTimeout 
        };

        var gRpcErrors = new StatusCode[] {
            StatusCode.DeadlineExceeded,
            StatusCode.Internal,
            StatusCode.NotFound,
            StatusCode.ResourceExhausted,
            StatusCode.Unavailable,
            StatusCode.Unknown
        };

        Func<HttpRequestMessage, IAsyncPolicy<HttpResponseMessage>> retryFunc = (request) =>
        {
            return Policy.HandleResult<HttpResponseMessage>(r => {
                
                var grpcStatus = StatusManager.GetStatusCode(r);
                var httpStatusCode = r.StatusCode;

                return (grpcStatus == null && serverErrors.Contains(httpStatusCode)) || // if the server send an error before gRPC pipeline
                       (httpStatusCode == HttpStatusCode.OK && gRpcErrors.Contains(grpcStatus.Value)); // if gRPC pipeline handled the request (gRPC always answers OK)
            })
            .WaitAndRetryAsync(3, (input) => TimeSpan.FromSeconds(3 + input), (result, timeSpan, retryCount, context) =>
                                {
                                    var grpcStatus = StatusManager.GetStatusCode(result.Result);
                                    Console.WriteLine($"Request failed with {grpcStatus}. Retry");
                                });
        };

        services.AddGrpcClient<CountryServiceClient>(o =>
        {
            o.Address = new Uri("https://localhost:5001");
        }).AddPolicyHandler(retryFunc);

        var provider = services.BuildServiceProvider();
        var client = provider.GetRequiredService<CountryServiceClient>();

        try
        {
            var countries = (await client.GetAllAsync(new EmptyRequest())).Countries.Select(x => new Country
            {
                CountryId = x.Id,
                Description = x.Description,
                CountryName = x.Name
            }).ToList();

            Console.WriteLine("Found countries");
            countries.ForEach(x => Console.WriteLine($"Found country {x.CountryName} ({x.CountryId}) {x.Description}"));

        }
        catch (RpcException e)
        {
            Console.WriteLine(e.Message);
        }
    }
}

but at the end WaitAndRetryAsync is never called.

I created a small project available on github in order to reproduce it.

My test is fairly simple. I start the client without a listening back-end, expecting to read 3 times the output from Console.WriteLine($"Request failed with {grpcStatus}. Retry"); on the console. But the policy handler in never fired. I have the following exception instead

Status(StatusCode="Unavailable", Detail="Error connecting to
subchannel.", DebugException="System.Net.Sockets.SocketException
(10061): No connection could be made because the target machine
actively refused it.

without any retry.

Answer 1

With the help of @PeterCsala I tried some fix.

As a first attempt I tried without DependencyInjection, registering the policy as follows

var policy = Policy
    .Handle<Exception>()
    .RetryAsync(3, (exception, count) =>
    {
        Console.WriteLine($"Request {count}, {exception.Message}. Retry");
    });

var channel = GrpcChannel.ForAddress("https://localhost:5001");
TestServiceClient client = new TestServiceClient(channel);

await policy.ExecuteAsync(async () => await client.TestAsync(new Empty()));

This way it's working.

Then I came back to DI and used to register the policy as follows

IAsyncPolicy<HttpResponseMessage> policy = 
Policy<HttpResponseMessage>.Handle<Exception>().RetryAsync(3, (exception, count) =>
{
    Console.WriteLine($"Request {count}, {exception.Exception.Message}. Retry");
});

var services = new ServiceCollection();
services.AddGrpcClient<TestServiceClient>(o => { 
    o.Address = new Uri("https://localhost:5001");
}).AddPolicyHandler(policy);

var provider = services.BuildServiceProvider();
var client = provider.GetRequiredService<TestServiceClient>();
    
var testClient = (await client.TestAsync(new Empty()));

And still not working.

At the end it seems AddPolicyHandler is not suitable for grpc clients?

Answer 2

This is not working for you because Retry is now built into Grpc. In order to make this work, register your service as follows:

var defaultMethodConfig = new MethodConfig
{
   Names = { MethodName.Default },
   RetryPolicy = new RetryPolicy
   {
       MaxAttempts = 3,
       InitialBackoff = TimeSpan.FromSeconds(3),
       MaxBackoff = TimeSpan.FromSeconds(3),
       BackoffMultiplier = 1,
       RetryableStatusCodes =
       {
           // Whatever status codes you want to look for
           StatusCode.Unauthenticated, StatusCode.NotFound, StatusCode.Unavailable,
       }
   }
  };
  var services = new ServiceCollection();
  services.AddGrpcClient<TestServiceClient>(o => { 
      o.Address = new Uri("https://localhost:5001");
      o.ChannelOptionsActions.Add(options =>
      {
          options.ServiceConfig = new ServiceConfig {MethodConfigs = {defaultMethodConfig}};
      });
  });

That will add the retry policy to your client. One other thing that you might run into. I didn't realize this at the time, but in my service implementation, I was setting up errors something like this:

var response = new MyServiceResponse()
// something bad happens
context.Status = new Status(StatusCode.Internal, "Something went wrong");
return response;

The retry logic will not kick in if you implement your service like that, you actually have to do something more like this:

// something bad happens
throw new RpcException(new Status(StatusCode.Internal, "Something went wrong"));

The retry logic you configured when registering your client will then work. Hope that helps.