Moq verifies in debug and fails sometimes in run

Question

I've developed a wpf-mvvm application that uses wmi to document what is built into a given Computer into a json file. I've used the mvvm model to develop the application and I am testing the viewmodels for there functions. So while writing Tests to check if a command properly executes I get a confusing error. I am trying to verify that no matter what I throw in the execute() method of the ICommand the mocked dialog and Mediator methods are hit.

What has me stumped is the fact that, when I debug the test, it definitely gets verified and I hit all the methods I want but when I just run the test, sometimes (around 40% of the runs) I get the exception that the setup was not matched.

The exact exception was this:

Moq.MockException : MockIMediator:217: This mock failed verification due to the following:

IMediator x => x.Send<bool>(It.IsAny<CreatePCCommand>(), It.IsAny<CancellationToken>()): This setup was not matched.

The code I am testing is the following:

private async Task CreateComputer(CancellationToken token)
{
    var task = Task<Computer>.Factory.StartNew(() => _factory.FromWMI());
    var (orderNumber, countryCode) = OpenDialog(); //Can be ignored since it's mocked to return default values.
    Computer computer;
    if (!string.IsNullOrWhiteSpace(orderNumber))
        computer = await task.ConfigureAwait(false);
    else
        return;


    if (await _mediatr.Send(new CreatePCCommand(orderNumber, countryCode, computer), token).ConfigureAwait(false))
    {
        var dialog = _ui.GetDialog("The PC was successfully documented.", "The PC was successfully documented.");
        _ = dialog.ShowDialog();
    }
    else
    {
        var dialog = _ui.GetDialog("The documentation of the PC failed.", "The documentation of the PC failed.");
        _ = dialog.ShowDialog();
    }

}

The test case is this:

[Theory]
[InlineData("")]
[InlineData("   ")]
[InlineData(null)]
[InlineData("Hello World")]
public void MainWindowViewModel_CreateComputerCommand_CallsTheSecondDialog(string input)
{
    _mediatr.Setup(x => x.Send(It.IsAny<CreatePCCommand>(), It.IsAny<CancellationToken>())).ReturnsAsync(true).Verifiable();
    _dialog.Setup(x => x.ShowDialog()).Returns(true).Verifiable();
    var vm = new MainWindowViewModel(_mediatr.Object, _notifier.Object, _cache, _view.Object,
      _computer.Object, _process.Object, _factory.Object);
    vm.CreateComputerCommand.Execute(input);
    
    Mock.Verify(_mediatr, _dialog);
}

What I've tried so far is changing the setup to either one of the following:

 _mediatr.Setup(x => x.Send(It.IsAny<CreatePCCommand>(), It.IsAny<CancellationToken>())).Returns(Task.FromResult(true)).Verifiable();
 _mediatr.Setup(x => x.Send(It.IsAny<CreatePCCommand>(), It.IsAny<CancellationToken>()).Result).Returns(true).Verifiable(); 

None of the above solved the problem and it pretty much remained the same as with the full test shown above. Still When I debug I can see that all the relevant lines are hit even with the other setups shon above as I've expected but when running it from the test explorer in Visual Studio 2019 with the run all command it sometimes just fails.

For versioning I am using Mediator version 8.0.0, Moq 4.16.1 and Xunit 2.2.0.

Has anyone ever seen something like this. Did I configure something wrong? Is there a blatant error in my Code?

Answer 1

When tests work on their own, but don't work when you run them all at once, it's usually a sign that they share some state (from test class members, static members, etc.).

In other words, Test1 corrupts the data of Test2.

This problem is one of the reasons justifying avoiding setup and teardown in tests.

We can see what appears to be fields in your tests (like _mediatr, _dialog, etc.), so this hypothesis seems plausible.

---

Workaround

You can sometimes work around this by disabling test parallelization. You can do that with [assembly: CollectionBehavior(DisableTestParallelization = true)] when using xUnit. You can check the xUnit documentation for more details.