Azure durable function on aks cluster not working

Question

I have created an azure durable function and put it in a docker container, everything is working in docker desktop, the azure durable function is using and MQ rabbit trigger. this the code:

        public async Task<List<string>> RunOrchestrator(
            [OrchestrationTrigger] IDurableOrchestrationContext context
            )
        {
            var job = context.GetInput<Job>();
            var outputs = new List<string>();

            try
            {
                job.JobCreationIdResult = await context.CallActivityAsync<string>("JobExecutor_CreateJobSimple", job);
            }
            catch (Exception ex)
            {
                _logger.ForContext<JobExecutorSimple>().Error(ex.Message, ex);                
                return outputs;
            }

            return outputs;
        }

        [FunctionName("JobExecutor_CreateJobSimple")]
        public async Task<string> CreateJob([ActivityTrigger] Job job)
        {
            _logger.ForContext<JobExecutorSimple>().Information($"result JobId: {job.MktJobId}");
            return "done";
        }
        
        [FunctionName("JobExecutor_RabbitMQStartSimple")]        
        public async Task RabbitMQStart(
            [RabbitMQTrigger("mkt-executor-q-local", ConnectionStringSetting = "mkt-Executor-RabbitMqConnection")] Job job,
            [DurableClient] IDurableOrchestrationClient starter)
        {            
            string instanceId = await starter.StartNewAsync("JobExecutorSimple", job);
            _logger.ForContext<JobExecutorSimple>().Information($"Started orchestration with ID = '{instanceId}'.");            
        } 

nothing fancy, with the same image deploy to aks I'm getting the following log:

[40m[32minfo[39m[22m[49m: Host.Startup[0]
      Job host started
[40m[32minfo[39m[22m[49m: Microsoft.Azure.WebJobs.Host.DrainModeManager[0]
      DrainMode mode enabled
[40m[32minfo[39m[22m[49m: Microsoft.Azure.WebJobs.Host.DrainModeManager[0]
      Calling StopAsync on the registered listeners
[40m[32minfo[39m[22m[49m: Host.Startup[0]
      Stopping the listener 'Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskListener' for function 'JobExecutorSimple'
[40m[32minfo[39m[22m[49m: Host.Startup[0]
      Stopped the listener 'Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskListener' for function 'JobExecutorSimple'
[40m[32minfo[39m[22m[49m: Host.Startup[0]
      Stopping the listener 'Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskListener' for function 'JobExecutor_CreateJobSimple'
[40m[32minfo[39m[22m[49m: Host.Triggers.DurableTask[0]
      Stopping task hub worker. IsGracefulStop: False. InstanceId: . Function: . HubName: SimpleFunction. AppName: . SlotName: . ExtensionVersion: 2.6.0. SequenceNumber: 2.
[40m[32minfo[39m[22m[49m: Host.Startup[0]
      Stopping the listener 'Microsoft.Azure.WebJobs.Extensions.RabbitMQ.RabbitMQListener' for function 'JobExecutor_RabbitMQStartSimple'
[40m[32minfo[39m[22m[49m: Host.Startup[0]
      Stopped the listener 'Microsoft.Azure.WebJobs.Extensions.RabbitMQ.RabbitMQListener' for function 'JobExecutor_RabbitMQStartSimple'
[40m[32minfo[39m[22m[49m: Host.Triggers.DurableTask[0]
      Task hub worker stopped. IsGracefulStop: False. Latency: 00:00:01.2084240. InstanceId: . Function: . HubName: SimpleFunction. AppName: . SlotName: . ExtensionVersion: 2.6.0. SequenceNumber: 3.
[40m[32minfo[39m[22m[49m: Host.Startup[0]
      Stopped the listener 'Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskListener' for function 'JobExecutor_CreateJobSimple'
[40m[32minfo[39m[22m[49m: Microsoft.Azure.WebJobs.Host.DrainModeManager[0]
      Call to StopAsync complete, registered listeners are now stopped
[40m[32minfo[39m[22m[49m: Host.General[337]
      Host lock lease acquired by instance ID '000000000000000000000000B7700D04'.

and basically the function doesn't do anything.

the first error that I was getting was relating to not being able to release the lease so I have to turn it off in the host file:

"extensions": {
    "durableTask": {
      "hubName": "SimpleFunction",
      "useAppLease": false
    }
  }

there is a way a azure durable function to work inside aks without KEDA or it is mandatory?, really we don't expect to scale.

Answer 1

Since you are not expecting it to scale you can deploy it directly to deploy to AKS . KEDA will only be required only if you are preempting a surge in demand for a small period of time and not much after that . Thus you don't need event driven auto-scaling .

To create a docker image for durable function please refer the below article :

Run a Durable Azure Function in a Container - Code it Yourself... (mendible.com)

To deploy your docker image to AKS refer the following documentation :

Kubernetes on Azure tutorial - Deploy a cluster - Azure Kubernetes Service | Microsoft Docs

You can also refer this Running Azure Functions on AKS – Robert te Kaat (wordpress.com)

Answer 2

I’ve recently spent a few weeks dealing with the same problem where DrainMode would kick in and stop the listeners from running for seemingly no reason, and only on AKS, we were not ever able to recreate it locally.

In our case the reason turned out to be that when the application was starting up, the base image was setting an environment variable ASPNETCORE_URLS to bind to port 80. However in AKS we were setting a security context on our deployment to not allow the container to run as root, and only root users can bind to ports < 1024. This led to the same log trace you have shared, where drain mode kicks in and the listeners stop, but with no logged reason for it to happen.

Our fix was to expose port 8080 in the dockerfile and set ASPNETCORE_URLS to http://+:8080