Gradle Build

[planetf1]

Ignore the 'current status' here & refer to update towards the end of this issue posted 9 June 2021

What is Gradle?

• Gradle is an alternative build tool to maven, which shares many of it's characteristics but with some key differences
• Gradle is used for large projects like android, and by companies including Netflix, LinkedIn, Adobe, Pivotal (spring)
• IDEs including Visual Studio Code, IntelliJ idea, eclipse are supported as well as tools we use in egeria including snyk, dependabot, sonar
• Gradle can still consume, and produce maven artifacts - meaning no change for consumers
• More about Gradle -> https://gradle.org

Specific pain points / Issues with Maven

• Maven is very focussed on building java projects. It's default project type is jar. This works well for building jars, but other languages/artifact types tend to be more problematic. In egeria we're becoming more diverse. Python scripts. notebooks, docker images, golang (for containers), multiple UIs including node. Whilst there are plugins for most of these integration tends to require much experimentation especially for multi platform. Gradle sets out to work with many languages
• Customization - maven has very strict rules around lifecycle, and customization is dependent on plugins and their customization. Ad-hoc tasks are tricky to add. For example launching a server to run tests was somewhat problematic.
• Maven activity has slowed. The benefit is stability, but there's less active community to discuss issues with. Mailing list posts, issues rely on very small number of contributors. Documentation is tough to work through, and much of it is not current
• As maven has evolved 'hacks' have been added to many plugins to support multi-module builds. With 320 modules and growing this makes some aggregate tasks very tricky to handle including java doc, site reporting
• The maven assembly plugin seems to not be able to layout the file structure we want - ie with server, server/lib, clients (inc. dependencies) - for example in maven trying to pull in dependencies with a moduleSet actually pulls in dependencies for ALL modulesets, making it tricky to get, for example, client only libraries laid out in the right place. - this was the 'final straw' that led me to investigate an alternative, though tasks prior have taken longer than one would hope, and there are further changes whilst possibly with maven are probably non trivial
• Our maven build is slow. We're pushing 30 minutes now, and whilst in an official build environment this may not be a problem - and we always have to build clean, code fix to full deployment takes a long time. Developers can often short-circuit this in the IDE, but as we move to more sophisticated mechanisms like dynamic connector loading this is going to be difficult to document/articulate whilst just doing a build is consistent/reliable and could be quick. We seem to be a LOT bigger than most maven projects as we have gone fine grained with our components
• Some lists of current issues relating to build improvements warnings failures - what we can see is quite a varied of lists of things it would be good to do. Much of this is around extensibility and flexibility. Maven is hard to adapt. Gradle benefits from more flexible scripting & community which should make many easier - though that is yet to be proven

Benefits to look at with gradle

• Incremental builds : A 'full' build only builds what has changed. Incremental compiles. This works more similarly to IntelliJ native compile
• Dependency Resolution aims to resolve to latest compatible versions with full conflict resolution, whilst maven looks for shortest path. Tends to mean many overrides, and more maintainance. Other dependency features include optional dependencies, api vs implementation etc
• More flexible technology integration - for example we have node to be added, also a go operator.

Comparison articles

• https://www.baeldung.com/ant-maven-gradle
• https://medium.com/@Colin_But/ant-vs-maven-vs-gradle-801fde21af80
• https://www.drdobbs.com/jvm/why-build-your-java-projects-with-gradle/240168608
• https://gradle.org/maven-vs-gradle/ -- a clearly biased perspective as written by the gradle team
• https://spring.io/blog/2020/06/08/migrating-spring-boot-s-build-to-gradle
• https://octoperf.com/blog/2019/07/25/kraken-gradle-multi-project-builds-for-maven-users/
• And a reverse move -> https://phauer.com/2018/moving-back-from-gradle-to-maven/

Other build tools

• https://bazel.build

What work is being done?

• This thread of work is to prototype and investigate whether gradle will work better for Egeria than maven.
• Code will be added to Egeria to support this prototyping, but could be removed if/when we decide to stop this investigation
• The list below represents a fairly full set of changes if we were to fully switch - will add more subtasks as they come up. Importantly this is EVALUATION at this point. The maven build remains definitive, supported, and will not stop or break.

FEEDBACK is very helpful. Feel free to comment below & contribute to discussion and code.

Running the gradle build

• Go to top level of the build tree
• Linux/macOS ./gradlew build
• Windows: gradle.bat build

Current Status

This is a list of the areas currently being explored. Note that a tick may mean coded in a fork and not yet in master

Basic gradle build

• Initial gradle build scripts & runnable & pass copyright checks
• Execute build in GitHub action
• Enforce Github action pipeline build passes on PR

Build publishing

• Publish POMs (requires updating each module's build file with more metadata)
• Generate assembly
• Publish to maven central (already do this for sas viya connector, need module definitinos)

Address missing or failing components

• server chassis spring app
• REST api docs
• ui chassis
• ranger/gaian components (these should be moved to a new repo anyway as per Move ranger code to new repo #458)
• Unit tests failing (this looks like a genuine egeria/maven issue, gradle gets it right...) - remove 'ignore errors'
• 2 broken FVT Gradle: Subject Area FVT fails #4903 (subject area) Gradle: data engine FVT fails #4904 (data engine)
• RAT type checks (license/copyright)
• docker build (thiis

Dependency Management

• Inherit version fixes from maven pom
• remove any hardcoded version references (from very early build)
• refactor to use native gradle dependency constraint definitions
• enforce enhanced dependency checks (similar to maven build ie on unused dependencies)

Build quality tools

• sonar (inc. jacoco code coverage) (this can probably use gradle - need to check)
• nexus IQ (this can probably use gradle - need to check)
• test reports (we don't have proper reports for maven either)

Checks

• Testing ie Notebooks etc all work as normal
• Compare gradle/maven build report - same dependencies? all components built

Documentation

• Update build tree docs
• Blog

[planetf1]

Added azure pipeline scripts for:

• verify : This will run on a PR. However it will be a non-blocking check for information only which I will monitor and disable if it causes any issues
• merge : this will run after code is merged into master or a feature/release

In both cases artifacts are not saved (yet) and the same build limitations as above apply. However it will help to ensure the gradle build keeps better aligned with any maven build changes, and keep an independent perspective on build quality.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[planetf1]

Assemblies now added

[planetf1]

Current deprecations to fix:

jonesn:egeria/ (gradle9) $ ./gradlew build  --warning-mode all                                                                              [8:20:28]
Starting a Gradle Daemon, 1 incompatible and 1 stopped Daemons could not be reused, use --status for details
The org.gradle.unsafe.watch-fs system property has been deprecated. This is scheduled to be removed in Gradle 7.0. Please use the org.gradle.vfs.watch system property instead. See https://docs.gradle.org/6.7/userguide/gradle_daemon.html for more details.

> Configure project :
The runtime configuration has been deprecated for dependency declaration. This will fail with an error in Gradle 7.0. Please use the runtimeOnly configuration instead. Consult the upgrading guide for further information: https://docs.gradle.org/6.7/userguide/upgrading_version_5.html#dependencies_should_no_longer_be_declared_using_the_compile_and_runtime_configurations
        at build_banbyu5zr3jyslv4nn2592cn0$_run_closure1$_closure4$_closure9.doCall(/Users/jonesn/IdeaProjects/egeria/build.gradle:177)
        (Run with --stacktrace to get the full stack trace of this deprecation warning.)

> Configure project :open-metadata-distribution:open-metadata-assemblies
The AbstractArchiveTask.extension property has been deprecated. This is scheduled to be removed in Gradle 7.0. Please use the archiveExtension property instead. See https://docs.gradle.org/6.7/dsl/org.gradle.api.tasks.bundling.AbstractArchiveTask.html#org.gradle.api.tasks.bundling.AbstractArchiveTask:extension for more details.
        at build_4c0obnkrn8g7h31rv54ltw1cj.run(/Users/jonesn/IdeaProjects/egeria/open-metadata-distribution/open-metadata-assemblies/build.gradle:208)
        (Run with --stacktrace to get the full stack trace of this deprecation warning.)
The AbstractArchiveTask.archiveName property has been deprecated. This is scheduled to be removed in Gradle 7.0. Please use the archiveFileName property instead. See https://docs.gradle.org/6.7/dsl/org.gradle.api.tasks.bundling.AbstractArchiveTask.html#org.gradle.api.tasks.bundling.AbstractArchiveTask:archiveName for more details.
        at build_4c0obnkrn8g7h31rv54ltw1cj.run(/Users/jonesn/IdeaProjects/egeria/open-metadata-distribution/open-metadata-assemblies/build.gradle:209)
        (Run with --stacktrace to get the full stack trace of this deprecation warning.)

> Configure project :open-metadata-test:open-metadata-fvt:access-services-fvt:asset-consumer-fvt
The testCompile configuration has been deprecated for dependency declaration. This will fail with an error in Gradle 7.0. Please use the testImplementation configuration instead. Consult the upgrading guide for further information: https://docs.gradle.org/6.7/userguide/upgrading_version_5.html#dependencies_should_no_longer_be_declared_using_the_compile_and_runtime_configurations
        at build_6k27ii8mxt5dqmr0ge5j5pvp2$_run_closure1.doCall(/Users/jonesn/IdeaProjects/egeria/open-metadata-test/open-metadata-fvt/access-services-fvt/asset-consumer-fvt/build.gradle:21)
        (Run with --stacktrace to get the full stack trace of this deprecation warning.)

[planetf1]

Deprecations fixed.. plus the build will now fail if deprecated function is used.

[planetf1]

Note on dependency resolution - see https://docs.gradle.org/current/userguide/resolution_strategy_tuning.html#sec:java_consistency for one approach to maintain consistency between compile & runtime dependency graphs

[planetf1]

In the community face to face 5-7 October 2022, it was agreed that we should continue to aim to complete this for release 4.x and look to apply this across all repositories (any old/stable repos we might leave?)

[planetf1]

Tips on maven->gradle migration, and equiv actions -> https://lmy.medium.com/how-to-convert-a-maven-project-to-gradle-in-real-life-2e3d5ee08650

[planetf1]

See #7007 for first pass to create maven artifacts

[planetf1]

Summary of differences in which artifacts are created maven vs gradle

Not in the gradle build

These are not present as they are 'packaging' poms, but were driven by the directory structure rather than a specific client use requirement for maven. In the gradle build we do have the artifact with only a pom - but no jar/sources/javadoc. This may fail maven central validation. In maven we padded them with 0 length jars.

Since these add no 'value' I think they should be removed

• access-services
• access-services-samples
• adapters
• admin-services
• admin-services-samples
• analytics-integrator
• analytics-modeling
• api-integrator
• asset-analysis
• asset-catalog
• asset-consumer
• asset-lineage
• asset-management-samples
• asset-manager
• asset-owner
• audit-log-connectors
• authentication-plugins
• catalog-integrator
• cohort-registry-store-connectors
• common-services
• community-profile
• configuration-store-connectors
• data-engine
• data-engine-proxy-services
• data-manager
• data-privacy
• data-science
• data-store-connectors
• database-integrator
• design-model
• design-model-archives
• dev-ops
• digital-architecture
• digital-service
• dino-view
• discovery-engine
• display-integrator
• egeria
• engine-host-services
• engine-services
• event-bus-connectors
• file-connectors
• files-integrator
• frameworks
• gaf-metadata-management
• glossary-author-view
• glossary-view
• governance-action
• governance-daemon-connectors
• governance-engine
• governance-program
• governance-program-client-samples
• governance-servers
• infrastructure-integrator
• integration-connectors
• integration-daemon-services
• integration-services
• it-infrastructure
• lineage-integrator
• metadata-security
• ocf-metadata-management
• open-connectors
• open-lineage-connectors
• open-lineage-services
• open-metadata-adapter-package
• open-metadata-archive-connectors
• open-metadata-archives
• open-metadata-caller-package
• open-metadata-client-package
• open-metadata-collection-store-connectors
• open-metadata-conformance-suite
• open-metadata-deployment
• open-metadata-distribution
• open-metadata-fvt
• open-metadata-implementation
• open-metadata-native-package
• open-metadata-packages
• open-metadata-resources
• open-metadata-samples
• open-metadata-test
• open-metadata-topic-connectors
• organization-integrator
• platform-services
• project-management
• repository-governance
• repository-services
• repository-services-connectors
• reset-client-connectors
• rex-view
• sample-metadata
• search-integrator
• security-integrator
• security-manager
• security-officer
• server-author-view
• server-chassis
• software-developer
• software-developer-client
• stewardship-action
• subject-area
• subject-area-client-samples
• tex-view
• topic-integrator
• ui-chassis
• user-interfaces
• view-services
• view-services-fvt

The following components are omitting source/javadoc in the gradle build

• data-privact-client -- we are missing source/javadoc in the gradle build
• data-science-client - as above
• design-model-client -
• dev-ops-client
• software-developer-client - no javadoc/source

Extra components in the gradle build

• graph-repository-connector --- also includes the jar-with-dependencies
• open-metadata-assemblies -- also contains the distrubution .tar.gz
• analytics-modeling-api also packages up analytics-modeling-${version)-tests.jar
• _remote.repositories in each component

Present as gradle produced maven artifacts, but not in the maven build ...

• access-services-fvt
• community-profile-fvt
• discovery-engine-fvt
• governance-engine-fvt
• governance-program-fvt
• open-types-fvt
• open-types-test
• open-types-test-generator

Build warnings
Packaging creates the following warning

Multiple publications with coordinates 'org.odpi.egeria:glossary-author-fvt:3.13-SNAPSHOT' 
are published to repository 'mavenLocal'. The publications 'connector' in project 
':open-metadata-test:open-metadata-fvt:access-services-fvt:glossary-author-fvt' and 
'connector' in project 
':open-metadata-test:open-metadata-fvt:view-services-fvt:glossary-author-fvt' 
will overwrite each other!

[planetf1]

May be useful to analyze gradle build performance/reliability. -> https://janbarari.github.io/gradle-analytics-plugin/

[planetf1]

Will revisit this after Mon 6 Feb when we fully switch main over (see #7330)

[planetf1]

Post-7330 the main checks will be

• recheck built artifacts are sufficiently similar
• check dependencies are up to date
• Look at fixing spdx header checks
• ensure all package warnings are fixed

[planetf1]

Re: the observations above

• the gradle additions seem to make sense
  • graph repo connector will be refactored into it's own repo in future
  • analytics modeling may need to be deprecated
  • having the .gz within the assemblies component has been useful in the past - it was removed to improve build performance
• additional gradle components vs maven were due to maven comparison not having run tests/fvts
• missing components created just to to directory structure for maven have no content. Discussed & agreed at face to face that these would not be included
• missing components with no source offer no value in any case. Effectively they are placeholders in the source to add the module later.

The remaining issue is spdx headers, which is already tracked by #6992

Closing