How can I pass tags when starting a cucumber android run using Gradle?

Question

I have a suite of cucumber android tests (they are instrumented tests) and I run them using gradlew connectedCheck. What I'm trying to do is run only certain tags when I run my tests (that's what tags are for, right?) using command-line options, without having to modify the code every time between test runs (so I can easily run these from CI server etc).

For example, I have tags @one and @two. I want to perform two gradle builds, the first with all the @one tests, and the second with all the @two tests. This would be part of an automated pipeline, so I wouldn't be able to modify the code (with the @CucumberOptions) between these two builds.

Here's what I've tried so far:

Attempt 1: Multiple CucumberOptions annotated classes

I tried having multiple runners (subclasses of CucumberAndroidJUnitRunner) and a different @CucumberOptions on each one, specifying different tags, then controlling which one was used via Gradle properies. I then did two builds, one with each runner. However, in both cases, Cucumber just used the @CucumberOptions annotation from the first runner alphabetically, so the tags were the same on both test runs. (In other words, even if the runner in use has a @CucumberOptions annotation, that's not necessarily the one that gets used.)

Attempt 2: Passing system property using gradle -D option

I found a hint from this thread that I might be able to pass -Dcucumber.options="--tags @two" but I tried this and it didn't work. It still just took the tags from the @CucumberOptions and ignored whatever I passed on the command-line. (This thread was confusing anyway as it started off talking about Gradle but later it was talking about Maven. I am using Gradle rather than Maven obviously).

I also tried -Dcucumber.filter.tags="@two" (which I found from the cucumber-jvm docs, talking about Maven not Gradle) but this also didn't work, the same as above.

Other investigations

• I looked at loads of online guides, but they were all based on either modifying the @CucumberOptions or running cucumber from the command-line (rather than via gradle)
• Obviously Java doesn't let you define annotation values by method calls, even if they are static methods, so the value in @CucumberOptions has to be constant (I double checked this)
• I could dynamically generate the @CucumberOptions-annotated class at runtime, but I couldn't figure out how to ensure that this happened before the cucumber instrumentation runner looked for it

Versions used

androidTestImplementation 'io.cucumber:cucumber-android:4.8.4'
androidTestImplementation 'io.cucumber:cucumber-picocontainer:4.8.1'

Answer 1

You can pass options through gradle using:

-Pandroid.testInstrumentationRunnerArguments.key=value

For example:

-Pandroid.testInstrumentationRunnerArguments.tags=@two

These options are then passed to instrumentation and handled by cucumber, see https://github.com/cucumber/cucumber-jvm/pull/597

So you can use for example:

gradlew connectedCheck -Pandroid.testInstrumentationRunnerArguments.tags=@two

Answer 2

For now, I am resorting to handling it myself through JUnit assumptions.

I added a new build config field based on a project property, with default value @main which is one of my tags.

android {
    defaultConfig {
        ...
        buildConfigField "String", "CUCUMBER_TAGS", "\"${getCucumberTags()}\""
    }
}

def getCucumberTags() {
    project.hasProperty("cucumbertags")
            ? project.getProperties().get("cucumbertags")
            : "@main"
}

I added a new class in my glue package

public class TagConfiguration {
    private boolean checkTagsMatch(final Scenario scenario) {
        // we check if at least one of the configured tags is present for this scenario
        // this is equivalent to Cucumber's "or" tag expression
        String[] configuredTags = BuildConfig.CUCUMBER_TAGS.split(" ");
        Collection<String> scenarioTags = scenario.getSourceTagNames();
        for (String configuredTag : configuredTags) {
            for (String scenarioTag : scenarioTags) {
                if (configuredTag.equals(scenarioTag)) {
                    return true;
                }
            }
        }
        return false;
    }

    @Before(order=1)
    public void assumeTagsMatch(final Scenario scenario) {
        Assume.assumeTrue("Only run scenarios that match configured tags", checkTagsMatch(scenario));
    }
}

This just makes it skip tests that don't match one of the tags - I can invoke like gradlew connectedCheck -Pcucumbertags="@one @two".

It's not ideal at all, as those tests still show up in reports but marked as skipped, and also my solution doesn't support tag expressions. If anyone has any better ideas, I would be interested to hear.