Azure durable function socket exception

Question

We have a durable function (v3) hosted on a consumption plan. This function uses Azure App configuration for the common settings of our software. The durable function calls Azure Luis to make tests on our model and improve it if necessary. This operation can be long (3 to 5 minutes) with many Http outbound calls.

Startup.cs

[assembly: FunctionsStartup(typeof(Startup))]
namespace FunctionTest
{
    public class Startup : FunctionsStartup
    {
        public override void ConfigureAppConfiguration(IFunctionsConfigurationBuilder builder)
        {
            builder.ConfigurationBuilder.AddAzureAppConfiguration(options =>
            {
                options.Connect(Environment.GetEnvironmentVariable(AzureAppConfigurationSettings.ConnectionStringsIdentifier));
                options.Select("*", Environment.GetEnvironmentVariable(AzureAppConfigurationSettings.LabelIdentifier));
            });
        }

public override void Configure(IFunctionsHostBuilder builder)
        {
            // We create the service provider to get the IConfiguration object from App configuration
            var serviceProvider = builder.Services.BuildServiceProvider();
            // We retrieve the service Provider as the startup must have a parameterless constructor
            var configurationRoot = serviceProvider.GetService<IConfiguration>();
            IServiceCollection services = builder.Services;
   var storageClient = new StorageClient(Configuration[StorageSettings.ConnectionStringIdentifier]);
 services.AddSingleton<IStorageClient>(storageClient);

            services.RegisterCosmosDb(
                Configuration[DocumentDBSettings.EndPointUrlIdentifier],
                Configuration[DocumentDBSettings.PrimaryAuthorizationKeyIdentifier],
                Constants.DocumentDB.DatabaseId, new List<string>() { Constants.DocumentDB.CollectionName, Constants.DocumentDB.QnATasksCollectionName });

 services.AddHttpClient<ILuisAuthoringService, LuisAuthoringService>()
                            // no retry policy because we have a custom one in the service in order to change Api when throttled
               .AddHttpMessageHandler<RequestLoggerDelegatingHandler>()
               .AddHttpMessageHandler<ResponseLoggerDelegatingHandler>();

            services.AddHttpClient<ILuisSubscriptionService, LuisSubscriptionService>()
            // no retry policy because we have a custom one in the service in order to change Api when throttled
            .AddHttpMessageHandler<RequestLoggerDelegatingHandler>()
              .AddHttpMessageHandler<ResponseLoggerDelegatingHandler>();

        }
}
}

StorageClient

public StorageClient(string connectionString)
        {
            ConnectionString = connectionString;
            CloudStorageAccount storageAccount = CloudStorageAccount.Parse(connectionString);

            
            TableClient = storageAccount.CreateCloudTableClient();
            TableClient.DefaultRequestOptions.RetryPolicy = new LinearRetry(TimeSpan.FromSeconds(1), 3);
            TableEndpoint = storageAccount.TableEndpoint;

            var tableServicePoint = ServicePointManager.FindServicePoint(storageAccount.TableEndpoint);
            tableServicePoint.UseNagleAlgorithm = false;
            tableServicePoint.Expect100Continue = false;
            tableServicePoint.ConnectionLimit = 100;
        }

And the extensions method that register azure CosmosDb

public static IServiceCollection RegisterCosmosDb(this IServiceCollection services, string serviceEndpoint,
            string authKey, string databaseName, List<string> collectionNames)
        {
            var documentClient = new DocumentClient(new Uri(serviceEndpoint), authKey, new JsonSerializerSettings
            {
                NullValueHandling = NullValueHandling.Ignore,
            }, new ConnectionPolicy()
            {
                ConnectionMode = ConnectionMode.Direct,
                ConnectionProtocol = Protocol.Https
            });
            documentClient.OpenAsync().Wait();

            var cosmosDbClientFactory = new CosmosDbClientFactory(databaseName, collectionNames, documentClient);
            cosmosDbClientFactory.EnsureDbSetupAsync().Wait();

            services.AddSingleton<ICosmosDbClientFactory>(cosmosDbClientFactory);

            return services;
        }

I inject this HttpClient like below in my service

public class LuisAuthoringService : ILuisAuthoringService {
        private readonly HttpClient _client;
      

        public LuisAuthoringService(
           HttpClient httpClient)
        {
            _client = httpClient;
        }
}

But the issue I have is that sometimes, when calling the status of my durable function I get the following error either from the startup (and I have a error "Function host is not running")

System.AggregateException : Retry failed after 3 tries. Retry settings can be adjusted in ClientOptions.Retry. (An attempt was made to access a socket in a way forbidden by its access permissions.) (An attempt was made to access a socket in a way forbidden by its access permissions.) (An attempt was made to access a socket in a way forbidden by its access permissions.) ---> An attempt was made to access a socket in a way forbidden by its access permissions. ---> An attempt was made to access a socket in a way forbidden by its access permissions. ---> An attempt was made to access a socket in a way forbidden by its access permissions.

  at async Azure.Core.Pipeline.RetryPolicy.ProcessAsync(??)

  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

  at async Microsoft.Extensions.Configuration.AzureAppConfiguration.UserAgentHeaderPolicy.ProcessAsync(??)
//Omited a part for brievity
  at async Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.LoadAll(Boolean ignoreFailures)

  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

  at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.Load()

  at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)

  at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()

  at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration()

  at Microsoft.Extensions.Hosting.HostBuilder.Build()

  at Microsoft.Azure.WebJobs.Script.WebHost.DefaultScriptHostBuilder.BuildHost(Boolean skipHostStartup,Boolean skipHostConfigurationParsing) at D:\a\_work\1\s\src\WebJobs.Script.WebHost\DefaultScriptHostBuilder.cs : 59

  at async Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService.UnsynchronizedStartHostAsync(ScriptHostStartupOperation activeOperation,Int32 attemptCount,JobHostStartupMode startupMode) at D:\a\_work\1\s\src\WebJobs.Script.WebHost\WebJobsScriptHostService.cs : 276
---> (Inner Exception #0) Azure.RequestFailedException : An attempt was made to access a socket in a way forbidden by its access permissions. ---> System.Net.Http.HttpRequestException : An attempt was made to access a socket in a way forbidden by its access permissions. ---> System.Net.Sockets.SocketException : An attempt was made to access a socket in a way forbidden by its access permissions.

or from my own service

System.Net.Sockets.SocketException
Exception while executing function: PublishLuis An attempt was made to access a socket in a way forbidden by its access permissions.
An attempt was made to access a socket in a way forbidden by its access permissions. 

From what I understand and see in the Azure Diagnostic panel of my function, I may have reach the maximum number of outbound connection, but what do I do wrong with my HttpClient ? Shouldn't the injection (even if its not static) use the same HttpClient twice ?

Thanks.

Edit: Added CosmosDbClient and TableStorage Client in code sample as they are also failing

Answer 1

I also had this exact problem. My solution was to make sure everything is resolved using .net dependency injection, and I removed all types of static classes which in my case was used by service classes similar to you LuisAuthoringService. I also only used the IHttpClientFactory as the injected class in my service classes constructors.

Startup.cs

public override void Configure(IFunctionsHostBuilder builder)
{
      services.AddTransient<ILuisAuthoringService, LuisAuthoringService>();
      services.AddHttpClient("yourApi",
           client =>
            {
                client.BaseAddress = new Uri("apiUrl.com");
                client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
            });
}

LuisAuthoringService:

public class LuisAuthoringService : ILuisAuthoringService 
{
        readonly IHttpClientFactory _httpClientFactory;
        readonly HttpClient _client;
          
        public LuisAuthoringService(IHttpClientFactory httpClientFactory)
        {            
             _httpClientFactory = httpClientFactory;
             _client = _httpClientFactory.CreateClient("yourApi");
        }
}

By using this pattern for all services classes in all functions i managed to get my code running fine in azure.