CASSANDRA-17750: Add docs for dependency management changes

[aratno]

No description provided.

[dcapwell]

don't need ,

[michaelsembwever]

i'm going to nit-pick a bit here… forgive me…

but i'm curious whether folk think this page would be more readable if it contains two separate unadulterated sections:

• Dependency Management 4.2+
• Dependency Management before 4.2

For example, the Changes to dependency management in Cassandra 4.2 section puts unnecessary cognitive load onto the reader if they just don't care about how it worked before 4.2
The reader just wants to know what to do.

[aratno]

I'm up for that - how's my last edit? This separates those two sections a bit more clearly. We can always edit based on feedback from more users as well.

[ekaterinadimitrova2]

I like the latest changes, I would probably just put before 4.2 first and then 4.2. POM file types shouldn't be only at the end I think or we need some kind of anchor to them maybe, where we refer to changes to be done to them or what changed in different versions.
Otherwise LGTM, this is more around how to organize to make the flow easier to follow.
Thanks for looking into that!

[aratno]

In my opinion, the main audience for this doc (in the short term) is developers who are familiar with how to add dependencies in the pre-4.2 world, and are trying to figure out how to do things post-17750. In that scenario, I think it makes sense to increase visibility around the changes by having that section come first. Over time, this will change as 4.2 becomes more familiar to more folks.

[michaelsembwever]

In my opinion, the main audience for this doc (in the short term) is developers who are familiar with how to add dependencies in the pre-4.2 world, and are trying to figure out how to do things post-17750.

I would disagree with this statement :) because that method only existed for a year, ref CASSANDRA-16391. There was a much longer precedence before (checking jar and license files into lib/). I do suspect most developers are next to starting from scratch (and we don't add deps that often).

If you want the extra explanation there, I don't object if it appears later. But I'm in the "don't explain, don't complain" camp, where a tl;dr what do i do comes first so I can be quick. (My example was overly concise, and only meant as an example.)

[ekaterinadimitrova2]

I am with Mick here. Also, I don't like the tribe knowledge which I am honestly even sure that it is not the case here. Not many people deal with the dependencies or do it regularly. Also, yes, good point, things changed last year. I prefer to be more noisy and things to be more documented than less.

[michaelsembwever]

Suggested change

	In Cassandra 4.2 and onwards, dependencies are managed in Maven POM templates
	in `.build/*-template.xml`. These templates are processed into valid Maven POMs
	with the `ant write-poms` task and copied to `build/*.pom`.
	The following POMs from `build.xml` have been migrated to templates:
	* `parent-pom` is now at `.build/parent-pom-template.xml`
	* `all-pom` is now at `.build/cassandra-deps-template.xml`
	* `build-deps-pom` is now at `.build/cassandra-build-deps-template.xml`
	To add new dependencies in Cassandra 4.2 onwards, add the dependency to
	`parent-pom-template` and `cassandra-deps-template`, and optionally
	`cassandra-build-deps-template` if the dependency is required for build only.
	See
	https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Management[the
	Maven docs] on how to reference dependencies in the parent POM from the child
	POMs.
	For dependency versions that need to be available in `build.xml` and
	`parent-pom-template`, specify the version as a property in `build.xml`, add it
	to the `ant write-poms` target, then add the property to `parent-pom-template`
	with the value of the template substitution.
	For more information on this change, see
	https://issues.apache.org/jira/browse/CASSANDRA-17750[CASSANDRA-17750].
	* Update dependency in `.build/parent-pom-template.xml` with version
	** Update `.build/cassandra-deps-template.xml` for runtime and test dependencies
	** Update `.build/cassandra-build-deps-template.xml` for build dependencies

[aratno]

I think it's important to spell this out in a little more detail, at least while the change is new. I'd support reducing it in the future, when people are more familiar with the process. But right now the change is so fresh that a newcomer might not get a good answer even after asking in Slack, and adding a little more detail here could make their lives a little easier.

[ekaterinadimitrova2]

I agree with Abe here. Even people long time on the project who don't deal with those things regularly will benefit

[Claudenw]

I think that the process needs to be spelled out because the people reading this will not be the developer that has experience with it. It needs to be clear in a fairly step by step process.

Linking to external documentation is advantageous and while I think we can expect that developers know how to specify a Maven dependency, the link to the Maven documentation should be included.

[michaelsembwever]

lines 5-9 need fixing. this sentence would be work after that.

[ekaterinadimitrova2]

Overall LGTM, my main point was around organizing the sections.
One thing on my mind is - should we add a note that new dependencies require a consensus on the mailing list here? Someone can miss it in the Code Style so I think it will be a good reminder here.

[ekaterinadimitrova2]

I like the latest changes, I would probably just put before 4.2 first and then 4.2. POM file types shouldn't be only at the end I think or we need some kind of anchor to them maybe, where we refer to changes to be done to them or what changed in different versions.
Otherwise LGTM, this is more around how to organize to make the flow easier to follow.
Thanks for looking into that!

[aratno]

One thing on my mind is - should we add a note that new dependencies require a consensus on the mailing list here?

Let's make those edits after the discussion here settles? https://lists.apache.org/thread/33dt0c3kgskrzqtp4h8y411tqv2d6qvh

I don't see anything in the code style page here about community consensus when adding dependencies: https://cassandra.apache.org/_/development/code_style.html

[aratno]

Just updated the intro based on feedback from @ekaterinadimitrova2 and @michaelsembwever. Could you approve if it feels ready?

[michaelsembwever]

One thing on my mind is - should we add a note that new dependencies require a consensus on the mailing list here?

Yes.

I don't see anything in the code style page here about community consensus when adding dependencies: https://cassandra.apache.org/_/development/code_style.html

See
https://lists.apache.org/thread/33dt0c3kgskrzqtp4h8y411tqv2d6qvh and https://lists.apache.org/thread/fwylw50s4nh393y79yz4z8k4gyj7ybvx

Hasn't moved from the gdoc yet: https://docs.google.com/document/d/1sjw0crb0clQin2tMgZLt_ob4hYfLJYaU4lRX722htTo/edit

[Claudenw]

I think that the process needs to be spelled out because the people reading this will not be the developer that has experience with it. It needs to be clear in a fairly step by step process.

Linking to external documentation is advantageous and while I think we can expect that developers know how to specify a Maven dependency, the link to the Maven documentation should be included.

[Claudenw]

This block does not seem to render correctly in the github viewer. That may be an artifact of the viewer or there may be a need for a blank line to separate the bulleted list from the proceeding paragraph.

However, I think the sentences should be changed to read:
X is now Y rather than X is now at Y as the later sounds like a file with the name Y will be found in the directory.

[Claudenw]

This block does not seem to render correctly in the github viewer. That may be an artifact of the viewer. However the * symbols are swallowed in the View File -> Preview mode.

[Claudenw]

The POM FIle Types section does not make sense where it is now. As it is at the same level as the 4.2 and pre-4.2 sections it would appear to apply to both, however, it mentions files that are only in the pre-4.2 code. I think that a section for the POMs found in each section would be cleaner and easier to understand.

[michaelsembwever]

see comments.

and… of curiosity… given the versioning differences in the doc, wouldn't this be better moved in-tree ? (easier to read, easier to maintain)

[michaelsembwever]

add an item about lib/ and resolver-dist-lib ?

[aratno]

Can you share more about what you mean? I'll admit I haven't used resolver-dist-lib directly so I'm not the right person to document its behavior for new contributors.

[michaelsembwever]

When you ant artifacts this will depend on the resolver-dist-lib target. This propagates the lib/ folder. This folder contains the jar files that are actually distributed in a release (e.g. the tarball, the deb and rpm packages). As opposed to build/lib/jars which contains more, e.g. build libs, provided scope, optional deps, etc.

We have to keep a closer eye on what gets generated into the lib/ folder, according to ASF distribution policy: https://www.apache.org/legal/resolved.html

[aratno]

I've added more details here

[michaelsembwever]

no you can't. nothing can be pushed public without a vote and three PMC +1s

(what ant publish does it stage it the ASF nexus, where we can then vote on it. but no one AFAIK needs to do this because we only do it as part of our release process.)

when a vote is finalised the staging repo in ASF nexus is "published" and gets pushed out to other public repos.

[aratno]

This was an existing part of the documentation, but I just made some clarifications - let me know if this is suitable for you

[Claudenw]

This is weird but the .build/\*-template.xml displays correctly on my system while the build/\*.pom does not. In my editor they both display correctly. Is there a way to see how the publishing system will render them?

[michaelsembwever]

see build instructions in README, and output of GHA.

[ekaterinadimitrova2]

Same notes as @michaelsembwever , except:

and… of curiosity… given the versioning differences in the doc, wouldn't this be better moved in-tree ? (easier to read, easier to maintain)

It is better if the instruction is in one place. Otherwise, people will be merging, breaking, and searching for more instructions... Keeping it in one place makes it easier. On the other hand, if we decide to split - this should be done in another ticket. This one is almost ready, and it has been reviewed three times already.

[aratno]

and… of curiosity… given the versioning differences in the doc, wouldn't this be better moved in-tree ? (easier to read, easier to maintain)

Agree with Ekaterina - I don't have a preference where this documentation lives. I'd just like it to be updated with information on the recent changes in how builds work.

[michaelsembwever]

I don't have a preference where this documentation lives. I'd just like it to be updated with information on the recent changes in how builds work.

That would be an argument to have it versioned :-)

No strong opinions here, the trade-offs I see are

• versioned: simpler to read, just go to the version you are working with and read half as much content. Less error-prone, and an easier page to maintain.
• unversioned: one place to learn how it works in all branches. more to read, more cognitive load. if working with multiple branches (e.g. forward merging) you have to do this anyway.

A question I have is if it's unversioned then what's happens to the docs on dependency version for branches that are no longer maintained. The code is not deleted but are we then deleting the documentation to it ? This isn't just archival purposes, we do still (in exception circumstances) apply CVE fixes to past branches (and the older branches are often those even the experienced devs need to refresh their memories on how things worked…)

(Even if we come to conclusion versioned is better, it does not need to be done in this PR.)

[aratno]

(Even if we come to conclusion versioned is better, it does not need to be done in this PR.)

I agree with your trade-offs listed. If you believe this structural change doesn't have to be done as part of this PR, do you need the discussion of pros and cons to happen here?

I've addressed the other feedback and would love for this to merge - we can always relocate docs as necessary afterwards.

[michaelsembwever]

do you need the discussion of pros and cons to happen here?

yes, totally make sense to me. it's input to what and where to do something that stems from and impacts here :-)

[michaelsembwever]

link is not working,

[aratno]

Oopsies - fixed

[aratno]

Here's my personal opinion:

I'm against any changes to documentation structure that are motivated by the minor changes in dependency management that started in 5.0. Right now there's a pretty straightforward path for a new contributor to find the contribution docs on cassandra.apache.org, and find their way to the dependency section. I'm not convinced that the changes in this PR exceed the threshold where restructuring makes things easier and not more complicated. It's not uncommon for a dependency change to impact multiple versions, so having both docs in one place feels very reasonable to me, especially while the change is relatively new. Using versioned documentation can make it harder to tell what's different between versions, which is relevant for contributors working with multiple release branches.

I'm not opposed to versioned dependencies in general, but it's more common for users to work with a single version than contributors, so defaulting to non-versioned docs for contributor-facing docs feels reasonable to me.

[michaelsembwever]

@aratno we're talking past each other here… 😃

Your involvement in this work makes your opinion super valuable and accessible. That's an opportunity to take advantage of 🙏 Creating a separate ticket to discuss the trade-offs risks creating a ticket unnecessarily, and losing the attention from the folks that probably have the most valuable opinion on it currently. Raising the discussion, having the discussion, deciding on an action, and where and when to do that action are all very separate tasks, there's no assumptions being made that doing some enforce others.

Appreciate your feedback! I generally agree with you, my concern remains about what happens to this documentation on EOL'd branches (there's benefit of just reading the accurate docs in -tree, rather than crawling through git log to find it…)

[aratno]

I think I understand your position better now - previously I had interpreted your comment as you prioritizing perfection over incremental progress, but now it's more clear that you find the discussion valuable even if it doesn't lead to an active effort immediately. I was maybe too wary of over-discussion - thank you for taking the step back to clarify.

Regarding EOL'd branches - aren't branches usually EOL'd when a new major comes out? Maybe we can do a docs sweep for every new major, as there's likely going to be a lot of other change happening to the docs at the same time?

[michaelsembwever]

Regarding EOL'd branches - aren't branches usually EOL'd when a new major comes out? Maybe we can do a docs sweep for every new major, as there's likely going to be a lot of other change happening to the docs at the same time?

Well yes, that's kinda my point. We'll possibly be cleaning up docs quickly (after each major release), and then those EOL'd branches no longer have the docs for them anywhere – and that's a problem, because our EOL'd isn't "nuke them", it's just we don't have the contributor resources to maintain these older branches but they still exist and others are free to step in and work with them. For examples: we do still apply CVE to older EOL'd branches as we see fit, and Microsoft has stepped forward and said they will provide support for older branches.

[aratno]

It seems reasonable to me that someone preparing a patch for an EOL'd version might have to dig a bit deeper and re-build the docs for when that version was supported. Having in-tree docs would make that easier, since their checkout of the EOL'd release would include the docs at that point in time, but we could also include a link to the SHA of the docs before they were pruned for that EOL. Then contributors would clone the EOL'd branch, find the ref for the docs, and be able to build those themselves.