multiproject reports using coverageAggregate are incomplete

[johnynek]

suppose you two subprojects: core and tests. Now, in tests, there are is some test harness code that makes it easier to run your tests that others might want (in tests/src/main). Then you exercise this harness in tests/src/test. The coverage for core/src/main that you get from the tests in tests/src/tests will be excluded from the report.

The report seems to only aggregate code that is tested IN ITS OWN test directory.

[lloydmeta]

👍 Running into this issue w/ a multi-subproject Play app as well.

In our case, we have integration tests written in the main/root app, where we inject concrete implementations for interface-typed constructor params in our subprojects' controllers. Like @johnynek , we are seeing that the coverage provided by these tests (in our root project) are not counted in the aggregated report.

cc @mauhiz

[kamilkloch]

+1

[bfil]

I'm experiencing this as well, I check regularly hoping for an update on the issue.

I tend to write libraries with the following multi-project structure:

example
example-testkit
example-tests

So providing a separate testkit can be used to write tests more easily for the users of the library, and the actual example-tests would also depend on the example-testkit to write the tests for the library.

This project structure just makes scoverage output 0% coverage on the main project, which makes it not very useful for this use case.

[kamilkloch]

Aggregated coverage seems to be working only after two consecutive runs of coverage test. See the demo of this peculiar behaviour here: https://github.com/kamilkloch/scoverage-multi-module

[bfil]

@kamilkloch you are right, the two consecutive runs do generate a usable report. The first one yields 0% test coverage but the second time it correctly generates the report.

I noticed though that using this approach generates broken links to the file sources in the report, the original issue about this was here: #69
The links are generated fine in the other case, but the coverage is wrong.

[kamilkloch]

@bfil My aggregated links are fine, are you using sbt-coverage 1.1.0?

[bfil]

Oh no that was my fault, I tried on a different project which was still on 1.0.4, on the projects using 1.1.0 the links are fine!

[RichardBradley]

I have the same problem, but running "test" twice doesn't fix it for me.

I have uploaded a sample project to https://github.com/RichardBradley/sbt-scoverage-multiproject-sample

If I run

sbt clean coverage test
sbt coverage test
sbt coverageAggregate

then I don't get any report for the "util" project: "No subproject data to aggregate, skipping reports"

Is there a workaround?

[RichardBradley]

I've found a workaround for my case:

sbt clean
sbt "set coverageEnabled := true" "set coverageEnabled in util := true" test coverageReport
sbt coverageAggregate

I found that running

sbt clean coverage compile

would instrument the main project but not the "util" project, but running

sbt clean coverage "set coverageEnabled in util := true" compile

would instrument the "util" project but not the main project.

EDIT: I'm testing if instrumentation was applied using "javap":

javap -c -classpath target/scala-2.10/classes/ myproject.App ; javap -c -classpath util/target/scala-2.10/classes/ util.Helper$

Something is definitely up with the "coverage" command in multi-project builds.

[RichardBradley]

I have a new version of the "coverage" command which fixes this for me:

commands += Command.command("coverageAllProjs", "enable coverage (all projects)", "") { state: State =>
  val extracted = Project.extract(state)
  val newSettings = extracted.structure.allProjectRefs map { proj =>
    coverageEnabled in proj := true
  }
  extracted.append(newSettings, state)
}

Now for the https://github.com/RichardBradley/sbt-scoverage-multiproject-sample project, if I run:

sbt clean coverageAllProjs test coverageReport coverageAggregate

then I get a working coverage report.

I'll create a PR to update the coverage command.

[sksamuel]

Thanks for your work on this Richard.

[RichardBradley]

kamilkloch: See the demo of this peculiar behaviour here: https://github.com/kamilkloch/scoverage-multi-module

That project seems to work for me for both ver 1.1.0 and my proposed 1.3.1 (from pull #129).
Was your problem already fixed, @kamilkloch ?

I'm running sbt clean coverage test ; sbt coverageAggregate

bfil: I check regularly hoping for an update on the issue.

@bfil -- does the fix in pull #129 work for you?
(Also: you do know you can click "subscribe" on a GitHub issue, rather than keep checking back right? ;-))

Thanks for your work on this Richard.

You're welcome -- I need this for my own projects, so it's mostly selfish :-)

[bfil]

@RichardBradley yep, I'm subscribed to it, just showing interest, anyway I will try it as soon as I can, thanks!

[pdalpra]

Argh, really sorry that I introduced that regression :'(

[kamilkloch]

@RichardBradley, unfortunately, scoverage 1.3.1 still does not work for me, now in two ways:

• coverageAggregate does not aggregate properly
• coverage command does not turn on coverage in subprojects (new in 1.3.1)

[RichardBradley]

@kamilkloch -- Please could you post a repro and describe the expected and observed behaviour?

The two behaviours you describe work for me with 1.3.1 in the sample project https://github.com/RichardBradley/sbt-scoverage-multiproject-sample as described above.

Thanks,

Rich

[RichardBradley]

Hang on; there's something wrong with the published 1.3.1 JAR.
Please see my comment on #129.
Please retest after that is resolved.

[oleg-gr]

I think it still doesn't work. Were there any changes since the last time it was brought up?

[gslowikowski]

I've just tested on @RichardBradley 's test project with version 1.3.5 and it works. Can you provide another test project?

[oleg-gr]

@gslowikowski it performs tests for all modules and aggregates them correctly (which I think is what the fix in this comment thread did), but I think it doesn't solve the original issue of one module using code from another module in its tests and getting coverage only for the code in its own module. Consider a project with Module A and Module B. If you write tests in Module A/tests that use both the code from Module A and Module B (Module A depends on Module B), you will only get coverage for packages and classes in Module A and not Module B. Does this make sense?

[gslowikowski]

This was discussed before. You need two runs. First sbt coverage test and second sbt coverageReport coverageAggregate. This is the only way to execute tests in your Module A before report generation for Module B.
Some users reported above that this does not work for them, but it works for me.

[oleg-gr]

@gslowikowski yeah, I was running sbt coverage test; sbt coverageReport coverageAggregate and I think it still doesn't take into account code from other modules for coverage report

[gslowikowski]

@oleg-gr I need test project

[oleg-gr]

@gslowikowski was creating a very simple project to reproduce the issue I was facing, but instead now running sbt clean; sbt coverage test; sbt coverageReport coverageAggregate doesn't generate anything at all. Any idea? https://github.com/oleg-gr/sbt-coverage-aggregate-sample-project

[gslowikowski]

@oleg-gr in your project you have Java classes, not Scala. Scoverage cannot instrument Java classes. Use Cobertura or JaCoCo.

[oleg-gr]

Makes sense. For some reason since I was using junit-interface I ended up thinking I should use sbt-coverage

[normana400]

this doesn't work in the scenario where the subproject has an inverted relationship
(ie. if the subproject 'utils' has a dependancy on the parent). in that case you cannot get coverage on an 'sbt utils/test' call

[RichardBradley]

[sbt-scoverage] doesn't work in the scenario where the subproject has an inverted relationship

Is that even allowed? It doesn't seem logically consistent for a sub-project to depend to the parent-project, it sounds like a circular dep to me (on the assumption that the sub-project is "part of" the parent project).

Why would you want that?

I suppose if SBT allows it, then sbt-scoverage ought to try to support it or (probably better) fail with a nice error message. How does sbt-scoverage currently behave in this case: does it give a helpful error or just silently do the wrong thing?

[VarnitaJain]

Does anyone know how to generate the report if some test case fails? what are the commands?

[gslowikowski]

@VarnitaJain - the same way as usual - sbt coverageReport or sbt coverageReport coverageAggregate for aggregated report in multi module project.

[kii-dot]

I'm encountering a similar problem for multi-module project.
my project structure is like this
`
parent

• app
• test
• modules
  • module1
    • src/main/test
• project
  `

however it only cover the tests that is in the parent folder, and it does not go into the modules folder at all.
I've ran this: sbt clean; sbt coverage test; sbt coverageReport coverageAggregate
and it still doesnt work. Is there a specific way multi-module should be structured?