How would I monitor and restart tasks that throw exceptions in C#?

Question

Let's say I have a program that instantiates three tasks that run indefinitely. These tasks are intended to run in parallel. However, lets say these tasks are known to occasionally throw exceptions due to network errors.

What would be the easiest technique for monitoring a task and restarting it if necessary?

My effort in solving this is to monitor the Task.Status data and simply call the Task.Start() method if the task has faulted.

However, this code doesn't work because an exception with the task causes the entire application to crash.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace ConsoleApp1
{
    class Program
    {
        static void Main(string[] args)
        {
            var my_task = Program.MainAsync();

            my_task.Wait();
        }


        public static async Task MainAsync()
        {
            var task_1 = Program.TaskMethod("1");
            var task_2 = Program.TaskMethod("2");
            var task_3 = Program.TaskMethod("3");

            // loop indefinitely restarting task if necessary
            while(true)
            {
                if (task_1.Status == TaskStatus.Faulted)
                    task_1.Start();
                if (task_2.Status == TaskStatus.Faulted)
                    task_2.Start();
                if (task_3.Status == TaskStatus.Faulted)
                    task_3.Start();

                await Task.Delay(1000);

            }

        }

        public static async Task TaskMethod(string task_id)
        {
            Console.WriteLine("Starting Task {0}", task_id);
            while(true)
            {
                await Task.Delay(5000);
                Console.WriteLine("Hello from task {0}", task_id);

                int i = 0;
                int b = 32 / i;

            }
        }

    }
}

Answer 1

Since a Task cannot be restarted, you could consider using task factories Func<Task> that can be invoked again and again every time a Task fails, to create more tasks.

List<Func<Task>> taskFactories = new();

taskFactories.Add(() => TaskMethod("1"));
taskFactories.Add(() => TaskMethod("2"));
taskFactories.Add(() => TaskMethod("3"));

Task[] enhancedTasks = taskFactories.Select(async factory =>
{
    while (true)
    {
        try
        {
            Task task = factory();
            await task;
            break; // On success stop
        }
        catch (OperationCanceledException) { throw; } // On cancellation stop
        catch { await Task.Delay(1000); continue; } // On error restart after a delay
    }
}).ToArray();

await Task.WhenAll(enhancedTasks);

Each factory is projected to an enhanced Task that includes the restart-on-fail functionality. The Select LINQ operator is used for doing the projection.

---

Note: The original answer that featured the Task.WhenAny-in-a-loop antipattern, can be found in the second revision of this answer.

Answer 2

I had also app crashes if a thread had an exception. So I learnt to always add try/catch to any thread or task method. Also the exception message could help to debug/improve the program.

I would just:

• use a try catch in the task methods
• use the exception as return value of tasks
• use a WaitAny not to poll too often, only react on a failure/end of a task
• use here a SortedDictionary to indentify the task and its start parameter

Here is the Code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace ConsoleApp1
{
    class Program
    {
        static void Main(string[] args)
        {
            var my_task = Program.MainAsync();

            my_task.Wait();
        }


        public static async Task MainAsync()
        {
            var tasks = new SortedDictionary<string, Task<Exception>>();
            for (int idx = 1; idx <= 3; idx++)
            {
                var parameter = idx.ToString();
                tasks.Add(parameter, Program.TaskMethod(parameter));
            }

            // loop indefinitely restarting task if necessary
            while (tasks.Any())
            {
                var taskArr = tasks.Values.ToArray();
                var finishedIdx = Task.WaitAny(taskArr, 30000);

                if (0 <= finishedIdx)
                {
                    var finishedTask = taskArr[finishedIdx];
                    var parameter = tasks.First(kvp => kvp.Value == finishedTask).Key;
                    if (finishedTask.Result != null) // exception was thrown
                    {
                        tasks[parameter] = Program.TaskMethod(parameter); // restart the task!
                    }
                    else
                    {
                        tasks.Remove(parameter);
                    }
                }
            }

        }

        public static async Task<Exception> TaskMethod(string task_id)
        {
            try
            {
                Console.WriteLine("Starting Task {0}", task_id);

                while (true)
                {
                    await Task.Delay(5000);
                    Console.WriteLine("Hello from task {0}", task_id);

                    int i = 0;
                    int b = 32 / i;

                }

                return null;
            }
            catch (Exception exc)
            {
                return exc;
            }
        }

    }
}

And then the output is like this:

Starting Task 1
Starting Task 2
Starting Task 3
Hello from task 3
Hello from task 1
Hello from task 2
Starting Task 2
Starting Task 1
Starting Task 3
Hello from task 2
Hello from task 1
Hello from task 3
Starting Task 2
Starting Task 3
Starting Task 1
Hello from task 2
Starting Task 2
Hello from task 3
Hello from task 1
Starting Task 3
Starting Task 1
Hello from task 2
Hello from task 1
Starting Task 2
Hello from task 3
...

Answer 3

I was working on a similar issue with "restarting" tasks that failed, and I tried Theodor Zoulias's answer, but it did not work for me because using the task itself as the dictionary key threw an exception in my case (the key already existed). I did use the answer as a starting point for my own solution. Here's something similar to what I came up with that uses a string to link the factories and tasks together:

var taskFactories = new Dictionary<string, Func<Task>>();

taskFactories.Add("task1", () => TaskMethod("task1"));
taskFactories.Add("task2", () => TaskMethod("task2"));
taskFactories.Add("task3", () => TaskMethod("task3"));

//Starts the tasks and stores them in a new dictionary.
var runningTasks = taskFactories.ToDictionary(x => x.Key, x => x.Value());

while (runningTasks.Count > 0)
{
    //Wait for something to happen, good or bad
    var completedTask = await Task.WhenAny(runningTasks.Values);
    string taskId = runningTasks
        .Where(x => x.Value == completedTask)
        .Select(x => x.Key)
        .First();

    if (completedTask.IsFaulted) //Something bad happened
    {
        Func<Task> factory = taskFactories[taskId];
        Task newTask = factory();
        runningTasks.Remove(taskId);
        runningTasks.Add(taskId, newTask);
    }
    else //The task finished normally or was canceled
    {
        runningTasks.Remove(taskId);
    }
}