Add Support to extract reports generated by Surefire/Failsafe/Jacoco Plugin

[shresthaujjwal]

What happened?

When you run mvn clean package, if you have surefire/failsafe/jacoco plugin (which includes test coverage), there isn't a way to view or extract those coverage reports using Maven Buildpacks

• What were you attempting to do?
  Trying to extract those coverage reports so that it can be feed into another code coverage tools so that Jenkins can
  https://plugins.jenkins.io/code-coverage-api/

Checklist

• I have included log output.
• The log output includes an error message.
• I have included steps for reproduction.

[dmikusa]

Can you provide some more context about this request, please? The default and typical use-case for buildpacks are to run and build an image, testing is specifically excluded. I think the general guidance is to execute tests through CI prior to performing a build, thus running tests a second time in the buildpack would be redundant.

You can of course change the maven arguments and run tests if that is your goal. You will see 100% of the output from Maven that Maven writes to STDOUT/STDERR. If you have Maven configured to write test output to the file system, that may not survive the build process (it depends where you write it) and best case it would end up in the resulting launch image as part of the artifact.

At any rate, I'm interested to hear more about how you're trying to use the buildpack here. If you could share more about your larger usecase, as well as provide some thoughts on how we could feedback the output from Maven to your other tools like Jenkins, besides STDOUT/STDERR which should presently work. Thanks!

[shresthaujjwal]

@dmikusa-pivotal Here is our normal CICD pipeline stages

• Checkout Code
• Build App
  • Build code using maven, gradle, npm, go etc. eg: mvn clean package (If you have surefire/failsafe/jacoco plugin, it will execute those plugin and generate code cov report)
• Analyze Code
  • Sonar Code Quality Gate. Here we upload the reports to Sonar. If code cov is low then build fails, code quality check is based on Sonar Quality Rules. https://docs.sonarqube.org/latest/user-guide/quality-gates/
• Deploy to Dev (You can ignore this step, we will have to create our own buildpack here that would deploy app to AWS)
• Acceptance Test Dev
  • Runs Dev Acceptance Tests using Cucumber or any other integration testing tools
• Release
  • Deploy to Artifact Repository Manager. Here we normally run mvn release and mvn deploy
• Deploy to QA (You can ignore this step, we will have to create our own buildpack here that would deploy app to AWS)
• Acceptance Test QA
• Performance Test QA (You can ignore this step, we will have to create our own buildpack here that would run PT)
• Upload Reports. Here we upload all our test reports from above acceptance test like cucumber reports to another internal application for compliance reason. We need those reports before can deploy anything to productions as it proves that we have done our due diligence to test our app in QA and all our cucumber test cases have passed.

So inorder for us to stay compliant we need access to all the reports generated by maven. So when we perform acceptance testing by running cucumber maven plugin it generates cucumber reports. It would be great to have access to this reports so we can feed this to other internal application
Hope this helps. Please share your thoughts.

[shresthaujjwal]

@dmikusa-pivotal just checking to see if you have any questions on this

[dmikusa]

I'm not sure how buildpacks can help with this one now. Extracting items from the container isn't really the intent of the buildpack.

In the step where you "upload reports", I could possibly see a buildpack performing this upload for you. i.e. run the tests, take the output and upload it somewhere, however, for it to be something that we'd pull into Paketo buildpack there would have to be general applicability of the technology. For example, a standard protocol, API or OSS project that would be used for the client/server in the uploads. Given that you mentioned it's an internal app, I don't think we could help there.

I think for this one, I would suggest we close it out. If you're interested, I would also suggest that you follow some of the planned work coming up with buildpacks. There are items on the roadmaps for both a develop API and test API. I haven't seen a lot of details on these, but I suspect they are going to be closer to what you want here. If you wanted to discuss these more or provided feedback, you could head over to the general channel on Slack and get connected with the team.

[loewenstein]

I think the general guidance is to execute tests through CI prior to performing a build, thus running tests a second time in the buildpack would be redundant.

Why, if you run Maven in CI already, would you want to have build tool buildpacks like Maven at all?

But anyway, I'd like to add to the discussion. We have built a CNB platform at https://www.project-piper.io/steps/cnbBuild that supports both mavenBuild followed by cnbBuild, potentially using the Maven build result from the target folder and cnbBuild directly on the sources.

The latter has the nice benefit, that there is no need for supporting all languages or runtimes in the CI/CD, but automatically supporting everything that the CNB ecosystem supports.

I'm not sure how buildpacks can help with this one now. Extracting items from the container isn't really the intent of the buildpack.

I could imagine that (reverse) bindings could be used to get information from the build to the platform, i.e. similar to a platform mounting a volume with Maven settings.xml to get config from platform to the build, the platform could mount a volume for buildpacks to provide build result related information back to the platform.

[dmikusa]

Why, if you run Maven in CI already, would you want to have build tool buildpacks like Maven at all?

The "general guidance" has really just been based on what we've seen users doing. Folks already have CI in place. Existing CI pipelines are already typically performing some exercise of the project's test suite. The least amount of work to integrate with buildpacks is to tack onto the end of the existing pipeline & use them for image generation. If the CI pipeline has already produced a compiled artifact and stored it somewhere, we can use that and make the image build even faster. If not, you can run the buildpacks against source code and we'll do that (also handy if you don't need to store/publish the built artifacts anywhere, i.e. they only end up in the resulting image). Since tests have already been run in the pipeline, we disable tests by default to reduce build times.

This is absolutely not the only way to do things. If you want to run tests during build you can do that. It's configurable through all of our build tool-based buildpacks. You should absolutely do whatever works best for your pipelines & business needs.

I could imagine that (reverse) bindings could be used to get information from the build to the platform, i.e. similar to a platform mounting a volume with Maven settings.xml to get config from platform to the build, the platform could mount a volume for buildpacks to provide build result related information back to the platform.

+1 - Seems like an interesting use case to me. If you want to exfiltrate things from the container, you'd need to raise this up with the buildpacks project. This isn't something that Paketo or other buildpack authors can do, we are limited to adding files to the resulting image. The tooling that drives buildpacks (i.e. pack, kpack, lifecycle, etc..) will need to support this type of process before we could do any sort of integrations.