mod/modfile: allow earlier language versions

[rogpeppe]

Currently when parsing in non-legacy mode, mod/modfile rejects any language version earlier
than v0.8.0, as that's when the language version field started to be supported and the
module syntax became closed, disallowing unrecognized fields.

@mvdan points out that it's useful for users to be able to specify a language version that's
earlier than that, as information about the actual language version that some code was
evaluated against.

We could see that the language version is before v0.8.0 and allow (but discard) any field
other than module, mirroring the fact that all fields were allowed before then.

[mvdan]

For some more clarity: particularly in terms of regression testing or Unity, it is a lot easier to understand the original writer's intended effect when knowing what version of CUE they were using. For example, even if the language spec didn't have any significant changes between v0.5.0 and v0.8.0, the evaluator still evolved behind the scenes, so it's very possible that their CUE code started behaving differently in some way - perhaps some errors are different, perhaps the exported result is different, or perhaps some errors appeared or disappeared.

I think it's fine if commands like cue mod tidy or cue mod get refuse to do anything until the user runs cue mod fix or manually updates the language version to one with a formal schema. We could say that module.cue files with old versions (before v0.8.0) are a form of legacy mode as well. But I also think it would be fine for us to apply the v0.8.0 schema to older versions, because it's very rare for downstreams to have extra fields other than precursors for language.version like cue.lang or cue.

[mvdan]

A passing thought: right now we are coupling the language version with the module.cue schema version. It's convenient to do this for us and for most users, because there is one less version to keep track of. But we don't need to tie the two together; someone might want to use the latest schema version, for example to be able to cue mod publish with a source.kind, but declare that the language spec version they are using is still v0.6.0. @rustamabd2 brought up this issue as one of our repositories had test module.cue files like:

module: "github.com/user_publisher/public_repo/mod/subdir@v0"
language: {
    version: "v0.8.0-alpha.3"
}
source: kind: "self"

which cmd/cue at master started rejecting as designed:

invalid module.cue file: source field is not allowed at this language version; need at least v0.9.0-alpha.0

The fix was to bump language.version, but in a way, it feels unrelated to update to a newer language spec version to be able to cue mod upload with a source (which is now mandatory, no matter what language version is declared).

Go has started splitting up the go version a bit; first with toolchain, and now with godebug: https://go.dev/issue/65573

I'm not saying we definitely want to split up language.version as well; I'm just saying we can, and Go has started doing so already. If we find that tying the two versions together is a pain for us or for users, we should consider it.

[mvdan]

Another example: right now we have some CUE users on ancient versions like v0.4.3 due to regressions or breakage in later versions of the evaluator. Once they do eventually update to v0.9.0 or v0.10.0, the current design would force them to set language.version to the latest version to be able to cue mod publish, whereas in general language spec jumps should be done very carefully and are unrelated to any modules features.

In this particular case, there aren't any breaking language spec changes between v0.4.3 and v0.9.0, but we do intend to make breaking changes in upcoming language versions: see for instance #2237 or #2932. If a large project makes use of language features that we are removing, a user should be able to stick to the older language.version for a bit longer until they are ready to make the jump.

[mvdan]

I also note that there are two possible solutions to the problem here:

1. Allow using ancient language.version strings like v0.4.3 by using a very lax module.cue schema for those.
2. Allow decoupling the module.cue schema version from language.version, by e.g. adding a new language.schemaVersion field which defaults to language.version, but can be higher than language.version when set.

Either of those will fix the immediate concerns, but longer term, 2 might be the better approach. If e.g. v0.11.0 ships with a new module.cue schema for cue mod publish features, but also breaking language spec changes, it would be unfortunate to force a user into updating both or neither at the same time.

Option 2 could also name the field something like "target version", which wouldn't be a maximum CUE version that can be used, but it would signal for example: even though I use the language spec at version v0.6.0, and that is the oldest version I support for evaluation, our developers use v0.9.0 for maintaining and publishing this module, and we want to make use of new backwards-compatible features like source.kind.

[rustamabd2]

Another thing is, trying to publish a module looking like this

module: "github.com/user_publisher/public_repo/mod/subdir@v0"
language: {
    version: "v0.8.0-alpha.3"
}

using cue tip version (v0.9.0-alpha.4.0.20240515162813-45755585913b) results in an error:

no source field found in cue.mod/module.cue

This seems wrong, as language version 0.8.0 didn't use the source field. Adding the source field changes the error to the one mentioned above and cue mod fix doesn't help with resolving this, either.

[mvdan]

The original blocking aspect of this for v0.9.0 was the fact that any modules created very early during the CUE modules experiment, such as one of mine with v0.8.0-alpha.3, started failing like:

cannot find schema suitable for reading module file with language version "v0.8.0-alpha.3"

This immediate issue got fixed by https://review.gerrithub.io/c/cue-lang/cue/+/1194685, where the oldest schema version was lowered from v0.8.0 to v0.8.0-alpha.0.

The other blocking bug I see here is what @rustamabd2 brings up in #3145 (comment); existing users of cue mod publish with a version like language: version: v0.8.0-alpha.3 might be very confused by this transition, as they are told to add a field that results in an error once added.

Those users on v0.8 were able to publish with v0.8 versions of the software, and now they cannot, unless they bump language.version to v0.9 and add source.kind. This seems like a hard regression, unless we are intentionally breaking those users. My understanding is that we should be backwards compatible and allow publishing those older modules as if they declared source: kind: "self", mimicking the old logic that wasn't aware of VCS at all. I will do such a change.

https://github.com/cue-labs/services/issues/141 seems to agree with my understanding by saying:

When language.version is at least v0.9.0-alpha.2, we know that the cue mod publish command fails if the source field isn't present, so we can do that same check in the registry too, predicated on language.version.

[mvdan]

https://review.gerrithub.io/c/cue-lang/cue/+/1195526 should fix this last issue brought up by Rustam.

We should leave this issue open for v0.10 for the sake of deciding if we want to allow setting very old language.version values, such as v0.4.3, which is important for Unity or other cases where an end user wants to declare "I wrote this code using ancient language spec semantics".

Earlier in this thread we also discussed decoupling the language version from the module.cue schema version. That's also a possibility, although not as much of an immediate need or issue as the others.

[mvdan]

As of July 2024, no users beyond me have asked for this feature - being able to declare an older language version like language: version: "v0.4.3". Supporting loading cue.mod/module.cue files with older language versions would also require some special logic to not enforce any strict schema.

The main use case I was thinking of here is Unity, where older projects in the corpus were written for older CUE versions, so I'd like to record that via language.version. However, those cue.mod/module.cue files from the original projects don't have the field, as back then it did not exist yet, so I would be adding it myself, for example as an overlay. I can always do that separately for the time being. It seems unnecessary to force this feature or compatibility mode if I'm the only user with Unity.

Closing for now; we can reopen if other users ask for this.