How to update Azure Devops test results with to display PassedOnRerun

Question

I am wondering how I can link my test rerun results to the original test results. My automation test pipeline looks like this:

1. run tests
2. publish results
3. check if there are failed tests
   • rerun failed tests if there are any
   • publish rerun test results

I would like to add the rerun test results to my original test run results so that I can filter tests on the pipeline using the "PassedOnRerun" toggle. Also if a test should fail the original and the rerun it would be nice to be able to have both failure reasons present in the original test run.

I'm unable to find a solid example of how to use the devops REST API to update a test result with a sub-result.

Answer 1

The same question has been posted here on MS Developer Community in 2020. The question got too few upvotes and therefore no technical answer.

As of May 2022 I can confirm that:

1. The problem still exists.
2. Publishing results of the same test with different PublishTestResults tasks NEVER reports "Passed on rerun", even if the second run contains the passing test with the same name and Id (no aggregation).

I also made these observations:

1. Publishing results with different PublishTestResults tasks reports each test execution as a SEPARATE test, even when same-named tests with same Id are present (no aggregation).
2. The TRX files published by a VSTest task with option "rerunFailedTests: True" ALWAYS produce "PassedOnRerun" (aggregation).
3. Publishing the very same TRX files from the above mentioned VSTest task using 2 separate PublishTestResults tasks NEVER produces "PassedOnRerun" (no aggregation).
4. There is no option for the PublishTestResults task that would produce aggregation.
5. There is no Azure DevOps REST API endpoint that would allow to mark test results as reruns, etc.

All these observations point clearly to the conclusion that the "Passed on rerun" reporting is a feature solely of the VSTest task and is not reproducible by another task. Have you tried using the VSTest task with option "rerunFailedTests: True"?

To have some reporting of the tests that pass on rerun, I use the following pipeline:

1. run tests (.NET Core task)
2. parse the produced TRX files
3. publish results ONLY for the passing tests (manipulate the TRX files)
4. run only the failed tests (parsed from the TRX files)
   • parse the newly produced TRX files
   • publish results ONLY for the passing tests - those are "passing on rerun"
   • publish results ONLY for the failed tests - those are "failing in 2 attempts"