Technology for workspace management and release tooling

[skaldarnar]

We've been on and off with discussing the need for more and better workspace tooling. This issue is intended to collect ideas and references regarding the topic to help us make an informed decision how to move forward.

Current Features

🚧 TODO: collect a list of features offered by groovyw ...

Requirements

🚧 TODO: figure out what we really need this for

References

• Tools for mono repo management like https://rushjs.io/ may already solve a couple of our problems, for instance publishing packages or checking for consistent versions
• Octokit to interact with the Github API. Official libraries for Ruby, .NET, and JavaScript, but lots of third party libs available for other languages. The launcher uses hub4j/github-api (kohsuke).

Multi-Repo Tooling

• Google's repo tool for managing multiple repositories within the same "workspace". Not necessarily something we should use directly, but we can take reference from their API design.

Gradle

There are also Gradle Plugins for semver version management (either via git tag or version file), like https://github.com/ethauvin/semver-gradle.

Maybe this could be integrated into the terasology-module Gradle plugin in some way, so we would use Gradle to perform this task... I don't know how clean the code for that would, though, and how well this works with git interaction.
And release plugins in general: https://github.com/researchgate/gradle-release

They usually work with version.properties or something similar, though.

CLI tooling

• CLIkt multi-platform command line parser for Kotlin
• kotlinx.cli seems to be another CLI argument parser for Kotlin
• http://yargs.js.org/ is a pretty neat library for writing CLIs in Javascript (auto-generating help messages, structure in sub-commands which automatically contribute to the overall CLI tool, ...)
• PicoCLI - any JVM?

POCs

• node-gooey. Side project by @skaldarnar to elaborate a CLI tool based on Javascript using yargs and octokit/rest.js
• gooky a Kotlin implementation of prepare-module-release script with CLIkt and JGit.
• GooeyCLI - next gen existing solution

[keturn]

Okay, I wrote a proof-of-concept version of the module-release script in Kotlin.

Just for fun, I decided to deliberately not look at the node-gooey implementation beforehand, to see how much our initial approaches would differ.

Here it is: https://github.com/keturn/Terasology/blob/spike/gooky/logistics/src/main/kotlin/gooky.kt

I find it highly amusing. "How to turn a 7-command script in to 200 lines of Kotlin."

Many of those lines are thanks to JGit. I didn't find any better Kotlin wrapper of a git interface, and I wasn't sure if the rules of engagement allow calling out to command-line git or not.

There are some really obvious opportunities for refactoring to cut the line count down, but I wanted to see what it looked like using the interface as given.

(Disclaimer: this is also not production code. You'll want more safeguards and more ergonomic error-handling than this rough version for something that's going to push directly to your protected branches.)

[keturn]

Oh, for usage:

I did this as a subproject in the multi-repo, because it's essentially a requirement for working in the multi-repo.

Do gradlew :logistics:installDist to build, and then you'll end up with the standard java-application-wrapper-scripts in logistics/build/install/gooky/bin

in real life, of course, we'd move those to someplace easier to type.

[Cervator]

I'm trying to resist wall-of-texting various design notes and feedback here until I can finally get out of Jenkins-migration-hell. But a real quick design highlight for one of the main bits of the groovyw approach that led to most the awkwardness in code: I was trying to support all sorts of different parts of the workspace, not only modules, but also facades (like the server variant), libs (crash reporter, etc), meta-modules (for Blender models, raw music, and such), and even had plans to support the wiki repos associated with each thing, and possibly a separately checked out home for gh-pages if we were to go that way to host micro-sites for things.

I'm not saying that's great design - i tend to architect further than my arms can reach, fully acknowledged. But a thought to refactor that in Groovy or build something better in something else would be high on my list. Along with how to actually run it, since it is hard to beat 1) Have Java (+ Git); 2) Clone; 3) groovyw something; 4) Run the game (gradlew jar game or cram into IDE).

Lightweight and powerful utility is really the name of the game here. Maybe the initial groovyw fit that, but growing it into something bigger could make it increasingly awkward and lumbering.

[skaldarnar]

(Disclaimer: this is also not production code. You'll want more safeguards and more ergonomic error-handling than this rough version for something that's going to push directly to your protected branches.)

This disclaimer also applies to my node-gooey 😅

I like how this is integrated with the engine repo, this is a huuuge advantage of this solution. Just works with the tooling we have installed anyways. I see that we put different focus, and also had different assumptions (I think we agreed on not using develop branches anymore, so everything would happen on master).

I think I could befriend Clikt, but I'm missing a library to handle Semver stuff in your PoC - is there none, or didn't you bother using one?

[keturn]

I used gestalt's Version class.

I expect we'd find something if we went looking, but I guess I'm not sure what we'd be looking for that's different than what org.terasology.naming.Version currently provides.

And no branch switching? Ah, that's why node-gooey has so much less git stuff! It's not only because it has a more concise git interface.

[skaldarnar]

@Cervator I could sketch out how to scale out the JS solution to support different target "objects", reusing common code, etc. Basically just a "cleaner" way of implementing the pattern gooey <scope> <command> we have for groovyw, too.

[keturn]

Other consideration: performance.

A lot of these tools aren't super performance-sensitive (...although users with mega-workspaces might have some feelings on that), but order-of-magnitude differences are worth noting.

• Right now groovyw usage in my environment takes about 5 seconds.
• after it's built, the basic gooky --help takes less than 0.5 seconds
• I don't have node-gooey checked out here, but doesn't have that many things in package.lock ("it's 800 lines!" yeah but that's small for npm stuff), so based on the performance of other node stuff I have here I'm guessing it's in a similar range to gooky.

[Cervator]

Alright - finally got something I can contribute. I made it in the form of a PR since it works as-is and would be ready to go if completed: #4065 - in short is is just next gen existing stuff, with far fewer quirks and way more power.

I added a couple lines to the top post here. I'm mostly just addressing the CLI stuff, less so the release management, but if "GooeyCLI" is the pick I expect a handful of lines would add the release management (same GrGit as other stuff), with a handful more to automate interaction with GitHub releases (need to hit the API, prolly @Grab Octokit). Haven't done that yet since :time: plus wanting to see how the release management itself really turns out

Performance-wise I hobo-stopwatched my way to just shy of 2 seconds running the basic gw help command for what it is worth.

Extra comments beyond the fairly meaty engine PR:

• At present it is again just loose Groovy files in the engine repo, although now they're actually on the classpath proper (fewer hacks!).
• They could be compiled with a tiny little @Graber script fetching the jar at runtime, to keep more code out of the engine, possibly run faster (already compiled), and way more importantly: could use it directly for DestSol as well
• No setup! No prep commands! Grab and go
• Extra use cases: beyond DS utility we might be able to use PicoCLI within the games as well? It seemingly supports most/all JVM languages, including Kotlin and Scala, even has examples
• Because I've managed to link this utility to a project I'm doing at work I can literally spend day job hours on making improvements to GooeyCLI (directly or indirectly). So I can about guarantee regular progress, at least for a while (unsure yet if it'll become a real boy product yet)
• I tried my best to set up the architecture for maximized reuse yet also make it really friendly and easy. I'd be thrilled for some feedback on what the other options do well (like the JS scaling). I'm not very confident in my basic Java-fu here. See the diagram in the PR

As for the "current features" and "requirements" here - well. Fundamentally it is about utility and quality of life enhancements for anything in the workspace. I could probably write a wall of text on detailing that, but I ended up just writing some code instead. I think the existing groovyw module init omega really speaks for itself (now gw init omega - shorter!)

I'd like to go for this option, but I'm not mandating it. Trying to make it stand on its own and do well in a comparison. What do you think, @skaldarnar & @keturn?

Even if I finish the current PR (without all the "future goals") it still won't quite address the release management, but I hope to get to that next.

[skaldarnar]

Thanks for the effort and update on this @Cervator.

At this point I'm leaning towards either of gw or gooky as they are closer to our ecosystem. I still haven't found the time to look at gw in detail, but the use of PicoCLI seems promising. Most of all, it addresses my main pain point of the Groovy scripts being really hard to maintain or extend. I may use node-gooey as little playground to test out some features or compare implementations and performance in a PoC style, but will focus on bringing one of the JVM solutions to life.

Without looking at gw in detail it's hard to say which one to prefer. Kotlin seems to be the "modern" choice, being a bit more trendy than plain old Groovy. It also comes with some nice language features, is backed by Jetbrains, and Gradle seems to have a future with Kotlin, too. We'd need to find a convenient way of bootsrapping it, though. That's where gw can shine, just using the whatever Groovy ships with Gradle. On the other hand, it comes with some Java boilerplate as far as I can tell. Is it the safe but boring alternative?

Let me try to turn this text into pro/con lists for both gooky and gw:

gooky (Kotlin)

• 🟢 fresh, modern language
• 🟢 backed by Jetbrains (we're already heavily using IntelliJ)
• 🟢 Gradle shifting to/supporting configuration in Kotlin
• 🟢 CLIkt library to aid CLI tool development
• 🔴 how to bootstrap a CLI tool

gw (Java/Gradle)

• 🟢 bootstrapping with Groovy wrapper (Groovy shipped with Gradle)
• 🟢 PicoCLI library to aid CLI tool development
• 🟢 many developers are at least familiar with some basic Java
• 🔴 Java boilerplate code
• 🔴 bound to Groovy version shipped with Gradle (currently v2, printing warnings...)

[Cervator]

Okay here's a quick stab at part 2: release management - but being that it is already midnight before a workday it is real brief (relative to my usual style)

I was able to spend a bit of time with TeraNUI, a brand new and pretty simple library of ours, thinking about and experimenting with the release management options. It mostly got stuck into an added https://github.com/MovingBlocks/TeraNUI/blob/master/gradle/common.gradle plus a pretty simple https://github.com/MovingBlocks/TeraNUI/blob/master/Jenkinsfile

Need to play with it some more and clean up (just realized the old isMaster method was left in), but in short it aims for simplicity yet makes fanciness available as needed. So it might qualify for the next phase approach for overall release management if applied to other things (with some more complexity - TeraNUI is easy in comparison)

In short: if version ends in -SNAPSHOT then publish will target a snapshot repo, and if not it'll target a release repo. No branch or Git checking of any sort. In the Jenkinsfile then there's a condition on the Publish stage on branch being master so CI builds of other branches and PRs won't cause published binaries. Whether it is ultimately master, develop, main or some other new name doesn't matter. It doesn't need the Git tagging approach to get involved either, but prepares for it (that'd be the next phase after)

You can trigger a release just doing the usual stripping of -SNAPSHOT from the version tag (in the root build.gradle in this case, bigger story there in other cases), commit, push to master. Bump to the next patch number, commit, push again to master and you'll get a new snapshot. If not setting to snapshot then successive pushes to master will cause build failures due to inability to re-publish the same release, as expected. But there is no more need for a funny module release management flag as there's no need for more than one branch or any needed differences between how modules handle their branches / releases differently (enabled or not)

I wouldn't go applying this to all the modules or anything yet though, not just from wanting to finish out the discussion, review, and so on, but also from wanting to see what GitHub does with default branch names. I'm curious to explore the Git tagging further, but that's sort of a different stage now.

• CLI automation - ready'ish, just not complete, or formally chosen yet
• Better release management with accurate jar files - about ready to start applying.
  • Gradle just does its thing in simple generic ways
  • Jenkins retains the hooks to do more things (publish, vary things for Nanoware builds, etc)
  • User manually triggers all releases via commits
• Actually triggering releases automatically is a next thing and has a bunch of scenarios I've tried to find a way to cover all of. I think I have.
  • Simple local releases (just push and CI handles)
  • GitHub releases, even fixing tags applied to snapshots
  • CLI utility command (gw module release x) -> handle commit plus tag for you, even the push ideally (need that for work anyway so I can spend time on it)
  • More fanciful ways like ChatOps or buttons but way lower priority
  • By including tag-based builds in Jenkins we can get good history and parallel release lines if needed (and by adding more conditionals skip builds that would otherwise trigger needlessly because both a commit, tag, and maybe even PR got involved at the same time) - yet this could also be entirely optional.

More later along with some more POC stuff ... 💤

[eviltak]

My 2 cents for versioning: enforce git tagging somehow; makes it really easy to check out the exact source of a specific version of a module/lib. This isn't applicable to snapshots of course, but if we really wanted to find a way to correlate snapshot releases to their sources, we could somehow add the 7-digit SHA value of the commit corresponding to the release in the version number/package metadata.

[skaldarnar]

Update: Added reference to repo tool from Android.