Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

code coverage for integration tests; make coverage more visible #317

Closed
cburroughs opened this issue Aug 24, 2017 · 3 comments
Closed

code coverage for integration tests; make coverage more visible #317

cburroughs opened this issue Aug 24, 2017 · 3 comments
Assignees
@cburroughs

Comments

@cburroughs
Copy link
Contributor

We currently generate code coverage information for unit tests but only generate a pretty report when the site goal is run. (So rarely, and not during the normal development workflow).

It would be better to:

  • (A) Always produce the report when tests are run (ie as part of the test goal). This appears straightforward.
  • (B) Generate reports for integration tests. This has not been straightforward.

To generate a report, jacoco needs access to the class files and reasonably expects them to be in java-manta-client/target/classes. However, because of (surprise!) our shade+relocation step the modified class files are only in the jar file. How jacoco keeps tracks of classes has its own page http://www.jacoco.org/jacoco/trunk/doc/classids.html and the interaction with other things that muck with bytecode is unhappy.

Options moving forward:

  1. Keep trying to make it work through maven. I can make a report manually (using the new jacoco cli), so it seems like it must be possible to do this somehow. But I don't have a good guess as to why what I have tried is any different from the cli. (I have also verified that things "Just Work" with shading disabled.)
  2. Give up for now and put some instructions in the readme for how to manually generate the integration test reports. I think in practice no one would look at this with any regularity.
  3. Move all integration tests to the main module. I think we are been burned by oddities of the shade+relocation often enough that there is real value in testing the same thing we are shipping.
  4. Some type of profile/conditional to make shading optional; CI runs integration tests twice. This might also give us an escape valve for Provide artifacts with no shaded dependencies #307
@cburroughs cburroughs self-assigned this Aug 24, 2017
@cburroughs
Copy link
Contributor Author

Some progress (internal):

Still missing source code linking for the integration tests, and the reports are not that useful without them. At first look, do the integration reports appear correct? I would have expected our integration coverage percent to by higher than our unit percent.

@cburroughs
Copy link
Contributor Author

After further hanging on option 1 above I have something that works. The above links should now have working code coverage reports. The workarounds are not super pretty but I think any problems would be confined to java-manta-it and not downstream users.

I would have expected our integration coverage percent to by higher than our unit percent.

I'm not sure what I was looking at before, but the integration test coverage is showing as higher, matching my intuition.

cburroughs added a commit to cburroughs/java-manta that referenced this issue Aug 29, 2017
@cburroughs
always produce coverage reports for unit & integration tests
597bcc8 
Previously code coverage reports were only generated as part of the
`site` goal, and were thus rarely looked at.  Binding the reports to
the same phase ensures they are always available for inspection.

Due to shading, several hoops need to be jumped through so that the
integration test report can refer to the source and classes under test.
Actions are bound to several maven phases to move files around, acting
as if java-manta-client was part of the integration test module.

ref TritonDataCenter#317
@cburroughs cburroughs mentioned this issue Aug 29, 2017
Merged
cburroughs added a commit to cburroughs/java-manta that referenced this issue Aug 30, 2017
@cburroughs
always produce coverage reports for unit & integration tests
d946c35 
Previously code coverage reports were only generated as part of the
`site` goal, and were thus rarely looked at.  Binding the reports to
the same phase ensures they are always available for inspection.

Due to shading, several hoops need to be jumped through so that the
integration test report can refer to the source and classes under test.
Actions are bound to several maven phases to move files around, acting
as if java-manta-client was part of the integration test module.

ref TritonDataCenter#317
cburroughs added a commit to cburroughs/java-manta that referenced this issue Sep 19, 2017
@cburroughs
always produce coverage reports for unit & integration tests
cbaa1ce 
Previously code coverage reports were only generated as part of the
`site` goal, and were thus rarely looked at.  Binding the reports to
the same phase ensures they are always available for inspection.

Due to shading, several hoops need to be jumped through so that the
integration test report can refer to the source and classes under test.
Actions are bound to several maven phases to move files around, acting
as if java-manta-client was part of the integration test module.

ref TritonDataCenter#317
cburroughs added a commit to cburroughs/java-manta that referenced this issue Sep 20, 2017
@cburroughs
always produce coverage reports for unit & integration tests
40ee31b 
Previously code coverage reports were only generated as part of the
`site` goal, and were thus rarely looked at.  Binding the reports to
the same phase ensures they are always available for inspection.

Due to shading, several hoops need to be jumped through so that the
integration test report can refer to the source and classes under test.
Actions are bound to several maven phases to move files around, acting
as if java-manta-client was part of the integration test module.

ref TritonDataCenter#317
cburroughs added a commit that referenced this issue Sep 22, 2017
@cburroughs
always produce coverage reports for unit & integration tests
b57de7a 
Previously code coverage reports were only generated as part of the
`site` goal, and were thus rarely looked at.  Binding the reports to
the same phase ensures they are always available for inspection.

Due to shading, several hoops need to be jumped through so that the
integration test report can refer to the source and classes under test.
Actions are bound to several maven phases to move files around, acting
as if java-manta-client was part of the integration test module.

ref #317
@cburroughs
Copy link
Contributor Author

Example runs (internal) after landing:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cburroughs cburroughs

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cburroughs