Added basic develocity setup #25322

[ribafish]

This PR will enable you to publish Build Scans to develocity-staging.eclipse.org as explained in the issue #25322.

Description

This PR publishes a build scan for every CI build (once CI credentials are created) and for every local build from an authenticated Eclipse committer. The build will not fail if publishing fails. Local and remote caching was left disabled on this PR by design so that the build is not affected by this change.

The build scans of the Eclipse Glassfish project are published to the Develocity instance at develocity-staging.eclipse.org, hosted by the Eclipse Foundation and run in partnership between the Eclipse and Gradle. This Develocity instance has all features and extensions enabled and is freely available for use by the Eclipse Glassfish project and all other Eclipse projects.

On this Develocity instance, Eclipse Glassfish will have access not only to all of the published build scans but also to other aggregate data features such as:

• Dashboards to view all historical build scans, along with performance trends over time
• Build failure analytics for enhanced investigation and diagnosis of build failures
• Test failure analytics to better understand trends and causes around slow, failing, and flaky tests

This will also enable you to (optionally) use build time optimization features, such as (remote) build caching and Predictive Test Selection.

More information can be read in the Eclipse announcement. This is the first part of Develocity integration issue no. #25322. The second part, if you opt for it, would be to enable caching for the project.

Please let me know if there are any questions about the value of Develocity or the changes in this pull request and I’d be happy to address them. If your project is interested in being a part of this initiative, please reach out to Eclipse infra by filling out a helpdesk ticket to get the CI credentials.

Locally, you can authenticate by running mvn develocity:provision-access-key. More info available here

[pzygielo]

@arjantijms can you link helpdesk ticket for secrets creation, please?

[pzygielo]

Unfortunately, as this change comes from non-committer the Jenkinsfile from target branch was used during the build for PR (and not the modified one).
And now, that modified Jenkinsfile comes from origin repository - it makes master to fail.

Now we need

• Fixed the wrong project id in Jenkinsfile. #25322 #25326
• and helpdesk ticket to create secrets.

[heurtematte]

If you wish to participate in the evaluation process, please open a ticket in the eclipse helpdesk.
We will provide the necessary bot/token, Jenkins plugins, and GitHub secrets to help you configure your CI.
Please note that this set up will require approval from a project lead.

[arjantijms]

@dmatej as the project lead, can you take a look?

[dmatej]

@pzygielo Can you create that helpdesk ticket? Or tell me what exactly do we need? Is it something like this?

I have doubts about the caching as the project is large and has significant technical debt and we already had issues with caching on Jenkins. Maybe it was be better to start the discussion with the community before merging it, and do some experiments to avoid blocking the whole project. But we already made this step, so lets resolve this and we will see.

What other leaders think about it? @hs536 @smillidge

[ribafish]

@dmatej To be clear, this PR only connects the project, caching (both local and remote) is left disabled, so that can be done separately.

As for the issue to get credentials: yes, this is exactly what is needed :)

I did run some tests with the local cache and got very nice results, though. Here is a scan from the experiment, where 54% of the time was saved + there are some tweaks I can do to further increase that number (and decrease the build time). Note that Develocity Build Cache works a bit different thatn Apache Maven Build Cache, as Develocity one works on a goal level, whereas Maven cache works on a project level. We have lots of experience caching large and complex projects, so I'd be happy to help out and resolve any issues that pop up.

[dmatej]

@ribafish Is it possible to disable the maven extension locally by default? My local builds slowed to 42 seconds from 36 after I rebased to current master. I tried several times, seems that the slowdown is visible - I do a lot of iterations, so any slowdown is visible in long term. I am not sure if the change has any effect on "full" builds, probably not as they are not so parallelized.

mvn clean install -Pfastest -T4C
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  42.218 s (Wall Clock)
[INFO] Finished at: 2025-01-17T18:04:55+01:00
[INFO] ------------------------------------------------------------------------
[INFO] 6282 goals, 6282 executed

[pzygielo]

And the documentation build is green.

I'd think that this should not be affected, https://ci.eclipse.org/glassfish/job/glassfish-website-publish-docs/1505/console, yet:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  04:45 min
[INFO] Finished at: 2025-01-18T19:31:04Z
[INFO] ------------------------------------------------------------------------
[INFO] 29 goals, 29 executed
[ERROR] Error executing a 'onClose' callback
java.lang.IllegalStateException: Failure creating Develocity storage directory
    at com.gradle.maven.common.configuration.j.i (SourceFile:79)
    at com.gradle.maven.extension.internal.dep.com.google.common.base.Suppliers$NonSerializab

and the site seems to remain not updated - https://glassfish.org/docs/latest/administration-guide.html:

[ribafish]

@pzygielo - This must be configured on the Jenkins instance, which is one of the reasons you need to open a ticket in the eclipse helpdesk. As for why it's affected - the extension is applied on the project level, which means that any build will have at applied. In order to publish scans, you need to have credentials configured, but if you don't, the scan just won't get published. This is an access error, which is why this build is affected.

@dmatej - how confident are you that this is because of Develocity? I can't reproduce it locally and I get varied build times(those are obviously with the Develocity extension) with no changes in between. Builds without it also vary from 2m35s to 3m06s, similar to the builds with the extension. Note: you can also set up publishing scans from local builds by authenticating to develocity-staging.eclipse.org by running mvn develocity:provision-access-key. That way you can use scans to troubleshoot what's going on, where is the time being spent.

[pzygielo]

In order to publish scans, you need to have credentials configured, but if you don't, the scan just won't get published. This is an access error, which is why this build is affected.

I'm completely not worried about scans being or not being published. But I'm bothered by updated documentation not being published to glassfish site and whole Jenkins PR build not working at all.

The build will not fail if publishing fails

As was said already -- in current state of Jenkinsfile -- the build is not even starting.

[ribafish]

@pzygielo I totally understand the issue here is CI is not working. To fix it, you (the project maintainers) need to open a ticket in the eclipse helpdesk so the Jenkins instance can be configured correctly.

[dmatej]

I totally understand the issue here is CI is not working. To fix it, you (the project maintainers) need to open a ticket in the eclipse helpdesk so the Jenkins instance can be configured correctly.

• I forgot to create the issue on Friday, I will do that now, sorry.
• The change blocked our CI - we can always revert the change, but maybe you can help us finding other options.
• The slowdown is well reproducible; I guess you cannot add extension collecting data over parallelized build without some price of synchronization. Also Glassfish uses own customized Maven lifecycle extension for glassfish-jar artifacts.
• Btw sometimes the extension produces some logs I don't understand, sometimes they are missing:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  48.426 s (Wall Clock)
[INFO] Finished at: 2025-01-20T11:30:35+01:00
[INFO] ------------------------------------------------------------------------
[INFO] 6282 goals, 6282 executed
led 20, 2025 11:30:36 DOP. sun.security.ssl.SSLLogger log
INFO: No available application protocols
led 20, 2025 11:30:36 DOP. sun.security.ssl.SSLLogger log
WARNING: Ignore impact of unsupported extension: server_name
led 20, 2025 11:30:36 DOP. sun.security.ssl.SSLLogger log
WARNING: Ignore impact of unsupported extension: ec_point_formats
led 20, 2025 11:30:36 DOP. sun.security.ssl.SSLLogger log
WARNING: Ignore impact of unsupported extension: session_ticket
led 20, 2025 11:30:36 DOP. sun.security.ssl.SSLLogger log
WARNING: Ignore impact of unsupported extension: renegotiation_info
led 20, 2025 11:30:36 DOP. sun.security.ssl.SSLLogger log
WARNING: SSLSocket duplex close failed. Debug info only. Exception details:

The last commit before merging this PR:

git checkout 9cd76c2945d55f47d88457ef21de3fc3111d75ea
mvn clean install -Pfastest -T4C
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  39.854 s (Wall Clock)
[INFO] Finished at: 2025-01-20T11:41:22+01:00
[INFO] ------------------------------------------------------------------------

[dmatej]

@pzygielo I see the log for the documentation ... I am not sure if credentials can resolve this, should we revert it?

[dmatej]

Issue in GitLab created: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/5546

[pzygielo]

I see the log for the documentation ... I am not sure if credentials can resolve this, should we revert it?

Unsure indeed - as it's separate job, not controlled by Jenkinsfile modified here. Perhaps the job has to reference secrets as well?

If so - which other jobs need that as well?

Maybe release job will also be not happy with

the extension is applied on the project level

and not being prepared for that.
We'll see.

[ribafish]

To expand a bit about what is causing the issue with the documentation above: The extension is working with some files in the user home folder, to which it expects to have the access to (by the user running the build). Since the Jenkins instance is configured so that the user running doesn't have permissions to modify those files, the build pipeline fails (this will get fixed now that there's an open ticket). It has nothing to do with credentials or secrets/vaults.

[ribafish]

@dmatej about the slowdown: We'd really like to troubleshoot this. Could you create a build scan of a build with the extension applied (for local builds you just need to authenticate once)?

Also, what do you mean by your own customized Maven lifecycle extension? Do we have access to it / could you link it?

There is a marginal overhead to capture the data and we do not need to synchronize in case of parallel builds, as we do most of the work on a specific worker thread.

[pzygielo]

I was wrong in #25324 (comment), as latest is apparently for latest release, but https://glassfish.org/docs/SNAPSHOT/security-guide.html is fine

and with develocity volume mounted the build https://ci.eclipse.org/glassfish/job/glassfish-website-publish-docs/1506/ completed w/o any trouble reported.

[dmatej]

@dmatej about the slowdown: We'd really like to troubleshoot this. Could you create a build scan of a build with the extension applied (for local builds you just need to authenticate once)?

Also, what do you mean by your own customized Maven lifecycle extension? Do we have access to it / could you link it?

Yes, here: https://github.com/eclipse-ee4j/glassfish-build-maven-plugin/blob/master/src/main/java/org/glassfish/build/GlassFishJarLifecycle.java
I don't know too much about ways how your software works, so I am just pointing to bit nonstandard part here.

There is a marginal overhead to capture the data and we do not need to synchronize in case of parallel builds, as we do most of the work on a specific worker thread.

I would still welcome a way to disable/enable it so it would be active only when we do want that (ie on CI).

[pzygielo]

This was

• reverted in Revert basic develocity setup #25335
• and reapplied in Added basic develocity setup #25322 #25336.

[ribafish]

@dmatej Hm, it doesn't look like the custom Maven lifecycle configures anything that would affect the performance when used together with the Develocity extension.

I created a script and Github Actions workflow to run some tests and I can confirm that with the Develocity extension, the average runtime when running mvn clean install -Pfastest -T4C is 13% slower on the GHA Ubuntu runner -> 2:31 min vs 2:52 min. After an investigation, it seems it's due to fingerprinting inputs time, which is a prerequisite of build caching. The picture changes, however, when cache is enabled, as the average build is 28% quicker for consecutive builds than no Develocity extension -> 2:31 min vs 2:08 min.

This speedup is even more visible when tests are executed (running mvn clean install), as the build is already ~56% quicker, without any further caching optimisations. Without the cache, average build takes 27:45 min (ignoring the ~14sec ones, which are failed) without the cache, while average with the cache takes 14min with the seeding build, with consecutive builds taking on average just 12:18 min.

To enable caching, I prepared 3 chained PRs:

1. Re-enable the Develocity integration, with a fix for the ant parallel jobs in the Jenkinsfile - cc @pzygielo
2. Enable local and remote caching without any optimisations
3. Enable caching and add caching for the asciidoctor goals