Could not load type Castle.Proxies.IReadinessProxy when running xUnit integration tests in parallel with Autofac

Question

I am having an issue that I've been many days unable to resolve. I use xUnit with a given-then-when abstraction to make tests more readable.

I am using a wrapper over EventStore and running some integration tests. They all go well.. except one that fails when running all in parallel (and xUnit runs in parallel), but if I run them all sequentially they all succeed.

I cannot understand why this would be an issue because every single fact is supposed to run the constructor (the given) and the functionality to test (the when). And in each fact I instantiate an Autofac ContainerBuilder, build the container and resolve its IComponentContext, so in theory every test should be isoalted and idempotent as intended.

This is the exception I keep receiving:

Autofac.Core.DependencyResolutionException : An exception was thrown while activating SalesOrder.EventStore.Infra.EventStore.EventStore -> SalesOrder.EventStore.Infra.EventStore.DomainEventsRetriever -> SalesOrder.EventStore.Infra.EventStore.Factories.DomainEventFactory -> λ:SalesOrder.EventStore.Infra.EventStore.EventTypeResolver.
---- System.Reflection.ReflectionTypeLoadException : Unable to load one or more of the requested types.
Could not load type 'Castle.Proxies.IReadinessProxy' from assembly 'DynamicProxyGenAssembly2, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'.
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters, Object& decoratorTarget) in C:\projects\autofac\src\Autofac\Core\Resolving\InstanceLookup.cs:line 136
   at Autofac.Core.Resolving.InstanceLookup.Execute() in C:\projects\autofac\src\Autofac\Core\Resolving\InstanceLookup.cs:line 85
   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters) in C:\projects\autofac\src\Autofac\Core\Resolving\ResolveOperation.cs:line 130
   at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable`1 parameters) in C:\projects\autofac\src\Autofac\Core\Resolving\ResolveOperation.cs:line 83
   at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance) in C:\projects\autofac\src\Autofac\ResolutionExtensions.cs:line 1041
   at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters) in C:\projects\autofac\src\Autofac\ResolutionExtensions.cs:line 871
   at Autofac.ResolutionExtensions.Resolve[TService](IComponentContext context, IEnumerable`1 parameters) in C:\projects\autofac\src\Autofac\ResolutionExtensions.cs:line 300
   at SalesOrder.EventStore.Infra.EventStore.Autofac.IntegrationTests.EventStoreExtensionsTests.ResolveTests.Given_A_Container_With_Event_Store_Registered_When_Resolving_An_IEventStore.When() in C:\src\SalesOrder.EventStore\SalesOrder.EventStore.Infra.EventStore.Autofac.IntegrationTests\EventStoreExtensionsTests\ResolveTests.cs:line 53
   at SalesOrder.EventStore.Infra.EventStore.Autofac.IntegrationTests.EventStoreExtensionsTests.ResolveTests.Given_A_Container_With_Event_Store_Registered_When_Resolving_An_IEventStore..ctor()
----- Inner Stack Trace -----
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.RuntimeModule.GetTypes()
   at System.Reflection.Assembly.GetTypes()
   at SalesOrder.EventStore.Infra.EventStore.Autofac.EventStoreExtensions.<>c.<RegisterResolvers>b__6_2(Assembly s) in C:\src\SalesOrder.EventStore\SalesOrder.EventStore.Infra.EventStore.Autofac\EventStoreExtensions.cs:line 174
   at System.Linq.Enumerable.SelectManySingleSelectorIterator`2.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
   at System.Collections.Generic.List`1.AddEnumerable(IEnumerable`1 enumerable)
   at SalesOrder.EventStore.Infra.EventStore.Factories.EventTypeResolverFactory.Create(IEnumerable`1 existingTypes, IReadOnlyDictionary`2 domainEventSerializerDeserializers) in C:\src\SalesOrder.EventStore\SalesOrder.EventStore.Infra.EventStore\Factories\EventTypeResolverFactory.cs:line 16
   at SalesOrder.EventStore.Infra.EventStore.Autofac.EventStoreExtensions.<>c__DisplayClass6_0.<RegisterResolvers>b__1(IComponentContext context) in C:\src\SalesOrder.EventStore\SalesOrder.EventStore.Infra.EventStore.Autofac\EventStoreExtensions.cs:line 180
   at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p) in C:\projects\autofac\src\Autofac\Builder\RegistrationBuilder.cs:line 62
   at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters) in C:\projects\autofac\src\Autofac\Core\Activators\Delegate\DelegateActivator.cs:line 71
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters, Object& decoratorTarget) in C:\projects\autofac\src\Autofac\Core\Resolving\InstanceLookup.cs:line 118

This is the test with only one Fact that fails when running in parallel with others:

public class Given_A_Container_With_Event_Store_Registered_When_Resolving_An_IEventStore
    : Given_When_Then_Test
{
    private IEventStore _sut;
    private IComponentContext _componentContext;

    protected override void Given()
    {
        var builder = new ContainerBuilder();
        builder
            .RegisterEventStore(
                ctx =>
                {
                    var eventStoreOptions =
                        new EventStoreOptions
                        {
                            Url = EventStoreTestConstants.TestUrl,
                            Port = EventStoreTestConstants.TestPort
                        };
                    return eventStoreOptions;
                },
                ctx =>
                {
                    var readinessOptions =
                        new ReadinessOptions
                        {
                            Timeout = EventStoreTestConstants.TestTimeout
                        };
                    return readinessOptions;
                });

        var container = builder.Build();
        _componentContext = container.Resolve<IComponentContext>();
    }

    protected override void When()
    {
        _sut = _componentContext.Resolve<IEventStore>();
    }

    [Fact]
    public void Then_It_Should_Not_Be_Null()
    {
        _sut.Should().NotBeNull();
    }
}

Any clue what may be happening here? I've had a look at Autofac's guidelines on concurrency: https://autofaccn.readthedocs.io/en/latest/advanced/concurrency.html but I think I am already using properly the component context.

UPDATE 1: FYI this is the GivenThenWhen template I use. But nothing very special here (I think!)

namespace ToolBelt.TestSupport
{
    public abstract class Given_When_Then_Test
        : IDisposable
    {
        protected Given_When_Then_Test()
        {
            Setup();
        }

        private void Setup()
        {
            Given();
            When();
        }

        protected abstract void Given();

        protected abstract void When();

        public void Dispose()
        {
            Cleanup();
        }

        protected virtual void Cleanup()
        {
        }
    }
    
    public abstract class Given_WhenAsync_Then_Test
        : IDisposable
    {
        protected Given_WhenAsync_Then_Test()
        {
            Task.Run(async () => { await SetupAsync(); }).GetAwaiter().GetResult();
        }

        private async Task SetupAsync()
        {
            Given();
            await WhenAsync();
        }

        protected abstract void Given();

        protected abstract Task WhenAsync();

        public void Dispose()
        {
            Cleanup();
        }

        protected virtual void Cleanup()
        {
        }
    }
}

UPDATE 2: And this is the IoCC Autofac registration's method I am using for my implementation and all the tests. A bit complex because I use reflection to keep the EventStore wrapper fully generic, but just in case somebody sees something funky there that might affect the tests..

public static class EventStoreExtensions
{
    public static void RegisterEventStore(
        this ContainerBuilder builder,
        Func<IComponentContext, EventStoreOptions> optionsRetriever,
        Func<IComponentContext, ReadinessOptions> readinessOptionsRetriever,
        Func<IComponentContext, CustomDomainEventMappersOptions> customDomainEventMappersOptionsRetriever = null)
    {
        RegisterEventStoreConnection(builder, optionsRetriever);
        RegisterFactories(builder);
        RegisterEventStoreManager(builder);
        RegisterConverters(builder);
        RegisterResolvers(builder, customDomainEventMappersOptionsRetriever);
        RegisterEventStoreServices(builder);
        RegisterEventStoreReadinessCheck(builder, readinessOptionsRetriever);
    }

    private static void RegisterEventStoreReadinessCheck(
        ContainerBuilder builder,
        Func<IComponentContext, ReadinessOptions> readinessOptionsRetriever)
    {
        builder
            .Register(context =>
            {
                var ctx = context.Resolve<IComponentContext>();
                var readinessOptions = readinessOptionsRetriever.Invoke(ctx);

                var timeout = readinessOptions.Timeout;
                var eventStoreConnection = context.Resolve<IEventStoreConnection>();

                var eventStoreReadiness =
                    new EventStoreReadiness(
                        eventStoreConnection,
                        timeout);

                return eventStoreReadiness;

            })
            .As<IEventStoreReadiness>()
            .As<IReadiness>()
            .SingleInstance();
    }

    private static void RegisterEventStoreConnection(
        ContainerBuilder builder,
        Func<IComponentContext, EventStoreOptions> optionsRetriever)
    {
        builder
            .Register(context =>
            {
                var ctx = context.Resolve<IComponentContext>();
                var eventStoreOptions = optionsRetriever.Invoke(ctx);

                ConnectionSettings connectionSetting =
                    ConnectionSettings
                        .Create()
                        .KeepReconnecting();

                var urlString = eventStoreOptions.Url;
                var port = eventStoreOptions.Port;
                var ipAddress = IPAddress.Parse(urlString);
                var tcpEndPoint = new IPEndPoint(ipAddress, port);

                var eventStoreConnection = EventStoreConnection.Create(connectionSetting, tcpEndPoint);
                return eventStoreConnection;
            })
            .As<IEventStoreConnection>()
            .SingleInstance();
    }

    private static void RegisterFactories(
        ContainerBuilder builder)
    {
        builder
            .RegisterType<DomainEventFactory>()
            .As<IDomainEventFactory>()
            .SingleInstance();

        builder
            .RegisterType<EventDataFactory>()
            .As<IEventDataFactory>()
            .SingleInstance();

        builder
            .RegisterType<ExpectedVersionFactory>()
            .As<IExpectedVersionFactory>()
            .SingleInstance();

        builder
            .RegisterType<IdFactory>()
            .As<IIdFactory>()
            .SingleInstance();

        builder
            .RegisterType<StreamNameFactory>()
            .As<IStreamNameFactory>()
            .SingleInstance();

        builder
            .RegisterType<RetrievedEventFactory>()
            .As<IRetrievedEventFactory>()
            .SingleInstance();
    }

    private static void RegisterEventStoreManager(ContainerBuilder builder)
    {
        builder
            .RegisterType<EventStoreManager>()
            .As<IEventStoreManager>()
            .SingleInstance();
    }

    private static void RegisterConverters(ContainerBuilder builder)
    {
        builder
            .Register(context =>
            {
                var utf8Encoding = new BytesConverter(Encoding.UTF8);
                return utf8Encoding;
            })
            .As<IBytesConverter>()
            .SingleInstance();

        builder
            .RegisterType<NewtonsoftConverter>()
            .As<IJsonConverter>()
            .SingleInstance();
    }

    private static void RegisterResolvers(
        ContainerBuilder builder,
        Func<IComponentContext, CustomDomainEventMappersOptions> customDomainEventMappersOptionsRetriever)
    {
        builder
            .Register(context =>
            {
                var ctx = context.Resolve<IComponentContext>();
                var customDomainEventMappersOptions = customDomainEventMappersOptionsRetriever?.Invoke(ctx);
                var domainEventSerializerDeserializers =
                    customDomainEventMappersOptions?.DomainEventSerializerDeserializers;
                var mapperResolver = MapperResolverFactory.Create(domainEventSerializerDeserializers);
                return mapperResolver;
            })
            .As<IMapperResolver>()
            .SingleInstance();

        builder
            .Register(context =>
            {
                var ctx = context.Resolve<IComponentContext>();

                var assembliesLoaded = AppDomain.CurrentDomain.GetAssemblies();
                var domainEventTypes =
                    assembliesLoaded
                        .SelectMany(s => s.GetTypes())
                        .Where(x => typeof(IDomainEvent).IsAssignableFrom(x)
                                    && x.IsClass);
                var customDomainEventMappersOptions = customDomainEventMappersOptionsRetriever?.Invoke(ctx);
                var domainEventSerializerDeserializers =
                    customDomainEventMappersOptions?.DomainEventSerializerDeserializers;
                var typeResolver =
                    EventTypeResolverFactory.Create(
                        domainEventTypes,
                        domainEventSerializerDeserializers);
                return typeResolver;
            })
            .As<IEventTypeResolver>()
            .SingleInstance();
    }

    private static void RegisterEventStoreServices(ContainerBuilder builder)
    {
        builder
            .RegisterType<EventStoreRepository>()
            .As<IEventStoreRepository>();

        builder
            .RegisterType<DomainEventsPersister>()
            .As<IDomainEventsPersister>();

        builder
            .RegisterType<DomainEventsRetriever>()
            .As<IDomainEventsRetriever>();

        builder
            .RegisterType<EventStore>()
            .As<IEventStore>();
    }
}

UPDATE 3: This is the whole repository in case somebody is bored and wants to try to reproduce it. It is a generic event store wrapper used for event sourcing implemented for Greg Young's Event Store product using the official C# driver.

https://gitlab.com/DiegoDrivenDesign/DiDrDe.EventStore

Funnily enough, this issue seems to disappear every once in a while. In fact, often, after restarting the PC all the tests pass properly. Other times it doesn't so I suspect it has something to do with the fact that I'm loading assemblies at runtime and something gets out of whack :(

---

UPDATE 9 Sep 2021 The async execution I was doing is not entirely correct. There is a way to embrace async execution with xUnit that may have been playing a role in this race condition. This is a better implementation of the given-when-then pattern all my tests inherit from. Thanks to IAsyncLifetime I am able to asynchronously (and await) the preconditions and the action itself.

using System.Threading.Tasks;
using Xunit;

namespace Rubiko.TestSupport.XUnit
{
    public abstract class Given_When_Then_Test_Async
        : IAsyncLifetime
    {
        public async Task InitializeAsync()
        {
            await Given();
            await When();
        }

        public async Task DisposeAsync()
        {
            await Cleanup();
        }

        protected virtual Task Cleanup()
        {
            return Task.CompletedTask;
        }

        protected abstract Task Given();

        protected abstract Task When();
    }
}

Answer 1

DynamicProxyGenAssembly2 is a temporary assembly built by mocking systems that use CastleProxy. There are some similar opened issues for NSubstitute and Moq that indicate that the problem is a race condition in Castle.Core or even in the .Net Framework (see: TypeLoadException or BadImageFormatException under heavy multi-threaded proxy generation for more details)

Answer 2

The inner exeption indicates that a type could not be loaded

Could not load type 'Castle.Proxies.IReadinessProxy' from assembly 'DynamicProxyGenAssembly2, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'.

and it came from this line :

at System.Reflection.Assembly.GetTypes()
at SalesOrder.EventStore.Infra.EventStore.Autofac.EventStoreExtensions.<>c <RegisterResolvers>b__6_2(Assembly s) in C:\src\SalesOrder.EventStore\SalesOrder.EventStore.Infra.EventStore.Autofac\EventStoreExtensions.cs:line 174
at System.Linq.Enumerable.SelectManySingleSelectorIterator`2.MoveNext()
at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()

which corresponds to this line in your code

var domainEventTypes = assembliesLoaded
                          .SelectMany(s => s.GetTypes())
                          .Where(x => typeof(IDomainEvent).IsAssignableFrom(x)
                                      && x.IsClass);

It means that one of the assembly contains a type that contains a reference to another assembly which is not loaded. Based on the name of the type in error it seems to be related with castle auto generated type.

You can subscribe to the static AppDomain.CurrentDomain.AssemblyResolve and AppDomain.CurrentDomain.TypeResolve events to better understand why the assembly is not loaded and maybe load it manually. See Assembly.GetTypes() - ReflectionTypeLoadException for more information.

In some case the exception is "normal" and could be ignored with a code like this :

public static IEnumerable<Type> GetLoadableTypes(this Assembly assembly)
{
    // TODO: Argument validation
    try
    {
        return assembly.GetTypes();
    }
    catch (ReflectionTypeLoadException e)
    {
        return e.Types.Where(t => t != null);
    }
}

Code from How to prevent ReflectionTypeLoadException when calling Assembly.GetTypes()

And you can use this here :

var domainEventTypes = assembliesLoaded
                          .SelectMany(s => s.GetLoadableTypes())
                          .Where(x => typeof(IDomainEvent).IsAssignableFrom(x)
                                      && x.IsClass);

Answer 3

If you have COM dependencies, make sure you set Embed Interop Types to No in your unit tests projects for these COM dependencies. By default, this property is set to Yes. When you mockup objects in your unit tests from these COM objects, they should not be embedded.

Answer 4

your table of database id DatabaseGeneratedOption.Identity and you try to save id of code you get error