Code Coverage fails to generate when instrumented tests combined with Cucumber

[fwpascual]

Description

I am having trouble combining Cucumber with Android Espresso tests and getting code coverage results. With code coverage disabled, the cAT gradle task completes successfully and with valid test results. With it enabled, however, the gradle task fails when generating the code coverage results.

I've created a sample project linked below that demonstrates the issue using the Android testing-samples repo as a baseline.

My issue seems similar to this issue, but I think it's a more advanced problem due to my usage of Cucumber. Using the unmodified sample from this tracked issue, I am able to get code coverage results:
https://issuetracker.google.com/issues/123987001

Steps to Reproduce

Using the linked repo below, you can see the issue by running
./gradlew clean cAT

You can download and build the provided repo that shows this issue directly; the following are steps that were used to create that project.

1. Downloaded the testing-samples project: https://github.com/android/testing-samples/tree/main/runner/AndroidTestOrchestratorWithTestCoverageSample
2. Added Cucumber to dependencies
3. Added a test file MyTest with CucumberOptions annotation
4. Added a SampleSteps with sample cucumber step
5. Added sample.features file with simple test
6. Run ./gradlew clean cAT with an emulator running

Expected Results

Expected a code coverage report

Actual Results

Get a failure during the :app:createDebugAndroidTestCoverageReport gradle task

* What went wrong:
Execution failed for task ':app:createDebugAndroidTestCoverageReport'.
> A failure occurred while executing com.android.build.gradle.internal.coverage.JacocoReportTask$JacocoReportWorkerAction
   > Unable to generate Jacoco report

Inspecting the folder under app/build/outputs/code_coverage/debugAndroidTest/connected/Pixel_4_XL_API_31(AVD) - 12, I see a Sample\ Feature#Sample\ Scenario.ec file, but it's zero bytes. When I look for this same file on the emulator, I see the same file under /sdcard/googletest/internal_use/data/data/com.example.android.testing.androidtestorchestratorsample/coverage_data/Sample Feature#Sample Scenario.ec with a 123 byte file.

Running the :app:connectedDebugAndroidTest task, I don't get a failure, and at this point the .ec file is also in the above described emulator device folder, however, I see this error in the gradle build log:

> Task :app:connectedDebugAndroidTest
Starting 1 tests on Pixel_4_XL_API_31(AVD) - 12
Mar 14, 2022 1:18:07 PM com.android.tools.utp.plugins.host.coverage.AndroidTestCoveragePlugin deviceShellAndCheckSuccess
WARNING: Shell command failed (1): cat "/sdcard/googletest/internal_use///data/data/com.example.android.testing.androidtestorchestratorsample/coverage_data//Sample\ Feature#Sample\ Scenario.ec" > "/data/local/tmp/b57c2a44-ba36-473d-9edf-5441bd14c514-coverage_data/Sample\ Feature#Sample\ Scenario.ec"
cat: /sdcard/googletest/internal_use///data/data/com.example.android.testing.androidtestorchestratorsample/coverage_data//Sample\ Feature#Sample\ Scenario.ec: No such file or directory

AndroidX Test and Android OS Versions

Emulator: Android API 31
Runner: 1.4.1-alpha03 (1.4.0 as well)
Test Services: 1.4.1-alpha03 (1.4.1 as well)

Link to a public git repo demonstrating the problem:

https://github.com/fwpascual/cucumber-coverage-sample

[brettchabot]

Sorry I'm not familiar at all with Cucumber - you might want to file this issue on cucumber's repository. I notice it uses a custom test runner 'io.cucumber.android.runner.CucumberAndroidJUnitRunner' which I would guess does not support orchestration.

Using a custom runner in general is not recommended for issues like this.

[fwpascual]

I have already logged an issue in the Cucumber repo around this same issue (cucumber/cucumber-android#96). I will pursue a resolution there but I was hoping you can maybe shed some light on why a workaround I devised is able to work.

I created a set of gradle tasks that will:

1. Delete all of the files in /build/outputs/code_coverage/debugAndroidTest/connected/{AVD} after the connectedDebugAndroidTest task
2. Do an adb pull /sdcard/googletest/internal_use/data/data/{package-id}/coverage_data to the cleaned out folder in step 1. This gradle task runs before createDebugAndroidTestCoverageReport

This implies to me that the .ec files are well formatted, however, either there is a problem with the timing of their creation or there is an issue with how these files are being copied to the local build folder. As mentioned in my initial issue, I'm getting a proper representation of .ec files matching the tests that were ran, but the copied over files are all zero bytes.

If you can please point me to the code that copies these files over, I can look to see if there is an incompatibility with what Cucumber's runner is doing and what is expected (e.g., check if there are restrictions on file names like no spaces or special characters).

[brettchabot]

@yuuki3655 can you provide some insight here?

[yuuki3655]

Sure. Those ec files are copied from Android device to a host machine after all test cases are finished by AndroidTestCoveragePlugin.

I downloaded your sample project, https://github.com/fwpascual/cucumber-coverage-sample, and I can reproduce the issue. However, after I upgraded the AGP version to the latest, "7.3.0-alpha07", it worked fine for me and it generated the coverage report HTML. Could you try the latest AGP version?

[fwpascual]

I can confirm that in my sample, updating to 7.3.0-alpha07 creates a valid coverage report. Thank you.

Applying the same change to my other project produces a new error, but I guess this is unrelated to this issue specifically.

java.lang.AssertionError: Activity never becomes requested state "[RESUMED, DESTROYED, CREATED, STARTED]" (last lifecycle transition = "PRE_ON_CREATE")
        at androidx.test.core.app.ActivityScenario.waitForActivityToBecomeAnyOf(ActivityScenario.java:338)

[austin226]

I can confirm that in my sample, updating to 7.3.0-alpha07 creates a valid coverage report. Thank you.

Applying the same change to my other project produces a new error, but I guess this is unrelated to this issue specifically.

java.lang.AssertionError: Activity never becomes requested state "[RESUMED, DESTROYED, CREATED, STARTED]" (last lifecycle transition = "PRE_ON_CREATE")
        at androidx.test.core.app.ActivityScenario.waitForActivityToBecomeAnyOf(ActivityScenario.java:338)

I got the same result when updating AGP to the newly released 7.2.0.