cmd/go, cmd/compile: record language version to support language transitions

[ianlancetaylor]

As part of moving toward Go 2 as initiated at https://blog.golang.org/toward-go2, I propose some changes to cmd/compile and cmd/go. For background see https://github.com/golang/proposal/blob/master/design/28221-go2-transitions.md.

I propose that we change cmd/compile to take a new -lang option. This option will take an argument which is the name of a Go language version. Go language versions use the same naming scheme as Go release versions and Go release build constraints: "goN.M". For example, -lang=go1.12.

This new option will direct the compiler to compile the code using the specified version of the language. It is an error to specify a future version of the language. All versions of the language from go1.12 forward must be supported.

Note that this may need to be recorded in the export data when describing any inlineable functions.

Since we don't know what language changes, if any, will be in the 1.12 release, it's hard to give a real example. But, suppose we had this feature for the 1.9 release, when type aliases were introduced to the language. Then passing -lang=go1.8 to the compiler would give a syntax error for any code that used a type alias. Passing -lang=go1.9 would permit type aliases as is usual today.

I further propose that we modify cmd/go to add a new directive to the go.mod file: lang goN.M. This directive sets the language version to use when building this module.

When updating a go.mod file, if the file has no lang directive a directive is inserted with the version of the go tool. Otherwise the lang directive is unchanged (unless of course go mod edit is used to change it).

When building a module, if the go.mod file has a lang directive, and it specifies a version N.M that is before or equal to the version of the go tool doing the building, then when invoking the compiler for this module the go tool will pass the option -lang=goN.M. In other words, if the module records that it was built using an older Go release, a newer Go release will build the module using the language version of the older release.

The rationale for these changes is explained in the design document mentioned above.

[gopherbot]

Change https://golang.org/cl/142417 mentions this issue: design: add Go 2 transitions background document

[beoran]

I like this idea.

I read the document and I would like to comment on it as follows:

I feel changes to the Go language that are not backwards compatible should increase the first version number of Go. I am normally not such a fan of semver but in this case it seems appropriate. In practise this will mean that after implementing all planned language changes, we will end up having a Go v5.x or maybe even v27.x, but that is fine, since it makes it very clear that the versions are definitely not backwards compatible.

I don't think redefinitions in the language are as much a problem as you make them out to be. In the case of the size of int changing, this was a backwards compatible change since the size of int is implementation defined, and the users of Go should not write any code that requires it to be of any given size. In this case, and all similar cases where backwards compatibility is retained in a redefinition, increase in the minor version of the is sufficient.

In the case of adding the key word check, this is in essence a backwards-incompatible change. This, as well as any other non backwards compatible changes should come with an increase in the major version number.

So if my proposal here is accepted, then we will certainly have a go version 2.0, soon to be followed by 3.0, maybe even 4.0 and 5.0. I think that the requirement to bump the major version number will give some pause as to whether or not we really want to make an incompatible change.

[mvdan]

I further propose that we modify cmd/go to add a new directive to the go.mod file: lang goN.M. This directive sets the language version to use when building this module.

I worry that this would be confusing, as go.mod already contains a go N.M statement. That isn't used for much yet, though, so perhaps this could piggyback off that statement. Edit: forgot to link to it: https://go-review.googlesource.com/c/125940/

I wonder if this would add lots of complexity over time to cmd/compile, as it would have to keep language support for an arbitrarily long number of Go versions. If a package uses a language feature introduced in 1.15, is a lang go1.15 statement in go.mod much better than the already possible go 1.15 statement?

[beoran]

GCC supports 5 different C++ standards from 98 up to 2a, and 3 different C standards from C89 until C11. Code that actually works is actually valuable, and there are many million line legacy projects that cannot be updated easily any more. Some complexity to stay backward compatible is unavoidable but necessary.

[ianlancetaylor]

@mvdan Unfortunately I haven't been closely tracking go.mod files. What does the go N.M statement mean and when is it updated? It doesn't seem to be documented.

Looking at the CL it doesn't seem to mean exactly what I'm looking for. It seems to be computed as the upper bound of the go N.M versions listed in required packages. Using those semantics would prevent a 1.20 package from depending on a 1.25 package while still using the 1.20 language version. I'm certainly fine with using the existing go N.M line if it will do what we need.

As the background doc says, it's definitely confusing that we use the same names for Go release versions and for Go language versions. I don't know how to avoid that. Fortunately I don't think the names will leak out to regular users very often.

I agree with @beoran that the additional complexity in cmd/compile is manageable.

[DeedleFake]

I really like this idea. A couple of random thoughts:

• Unlike module versions, this should probably allow automatic patch upgrades. For example, if I said lang go2.2, it would always use the latest supported v2.2.x.
• Go 2 and on would need to require a go.mod file to specify the version, or otherwise existing code that hasn't been upgraded to a module would quickly stop working or, depending on what changes were made to the language, silently start working differently.

[magical]

1. go.mod is a convenient dumping ground for this kind of metadata, but it's not obvious to me that module scope is the right granularity for the lang directive. It seems like it would be more useful at package or even file scope, to allow large projects to transition gradually/piecemeal.

2. You mention Perl 6 and Python 3 as examples of what not to do. It might also be worth discussing Python's __future__ imports and Perl 5's use VERSION as more successful examples of how to evolve a language.

[DeedleFake]

I don't think __future__ solves the same problem. The point of __future__ is to allow people to start using features that haven't been released yet and allow testing of those features. The point of this proposal is to allow a Go 2 project to import a Go 3 project which imports a Go 1 project and have it just work. That way legacy code doesn't need updating when backwards incompatible changes get made. This should hopefully allow Go to avoid the migration problem Python has had.

[magical]

It's certainly a different approach, but i do think it tackles the same problem: namely, "how can we add language features without breaking old code?".

Instead of asking old code to tag itself with an old language version, as in the current proposal, Python asks new code to explicitly request the features it needs. There's no reason a priori why this approach couldn't be adopted by Go (although i agree that it probably wouldn't be a good fit).

For one thing, in Python, the transition period during which old code and new code can be mixed is usually only a single release, after which the feature becomes the default; whereas Go would presumably want to extend this for a much longer period of time, possibly indefinitely. In such a case, one can imagine that feature directives would tend pile up in new code, leading to an unergonomic language, which is why i say it probably wouldn't be a good fit for Go.

In any case, it seems worth discussing in the proposal, if only to reject.

[ianlancetaylor]

@DeedleFake I am assuming that we are never going to change the language in a minor version, so lang go2.2 is going to mean exactly the same thing for 2.2.0, 2.2.1, 2.2.2, etc. If that assumption is wrong, then we may need to use three digits in a language version.

I'm not sure it's precisely accurate to say that code without a go.mod file containing a lang directive will quickly stop working. It would only stop working if we removed some language feature that that code relies on. But that is going to be a rare step. It's an important case to consider, but we aren't going to remove features that are widely used. That said, you are correct that using the go.mod file assumes that those will become widely used. If that is not the case, we should record the maximum language in some other way.

Which is, of course, what @magical suggests. I'd prefer to use the go.mod file if possible, because every separate piece of metadata is a significant pain point. Even if go.mod is not ideal, I think it's worth using if we can, just to avoid that pain.

@magical I updated the background document CL to include a section on Python and Perl feature selections. I don't think feature selection is a good fit for Go, but I agree that it's worth mentioning.

[DeedleFake]

Maybe a change to the package statement? Could be something like package main go2, for example, and a lack of a second argument would be the same as saying go1. That would increase the granularity and provide a place to quickly check what version you're dealing with. Editors and plugins could then detect the version of existing files when creating new files, and copy it, the way vim-go does already for the package name. One downside, though, would be that updating could be more awkward, as you'd have to change every file in the package.

Personally, I think go.mod would probably be a better fit, but I figured I'd throw this in there.

[networkimprov]

more useful at package or even file scope

Building on @magical's insight, here's a //go: directive listing only backward-incompatible language features used in the file.

//go:lang handle, xyz

That's more descriptive than min and/or max versions. (Does a max version imply dependence on the bugs or performance of that version?) This directive (or its absence) also informs new versions of the compiler to disable backwards-incompatible features when compiling older code.

One line should be enough, since very few backwards-incompatible features are expected. A //go:lib directive could do the same for the stdlib.

[ianlancetaylor]

Any annotation in the source code itself is troubling because we ideally want the go tool to be able to record the version being used, one way or another, but we certainly do not want the go tool to modify people's source code.

[networkimprov]

go.mod could store the tool version that was selected via a feature directive or the module author. That is build metadata, whereas a list of backwards-incompatible features required by a specific file is code metadata.

EDIT:
I think the main drawback to a //go:lang directive is new boilerplate for Go2 code.

[rsc]

go.mod already recognizes a 'go 1.8' etc line. I think we should use that instead of 'lang go1.8', and have the language apply to the whole module.

I'm on board with cmd/compile accepting -lang and whatever changes are needed to make compilation with one language version be able to import compilations from another language version. (I would hope that exported information would just be written in some 'go superset' instead of having to tag individual packages during export with language versions.)

[ianlancetaylor]

@rsc using the existing go line in go.mod files may be OK. (I didn't know about it because it isn't documented.)

The current go line does seem to have a somewhat different meaning than I was envisioning. Right now it seems that if a go.mod file says go 1.21, and I am building with Go version 1.20, I will get an error saying "module requires Go 1.21". What I am proposing here is that that case should just compile without incident. I think it's important that the language version be a maximum required language version, not a minimum. Otherwise, everyone is forced up to the version of Go used by whoever built the newest library that they depend on. That's a fast pull to tip, which is not going to work for organizations that need to be conservative. Pulling to tip is not required in the general case, and we shouldn't require it here.

[rsc]

Right, we need to figure that out - we've started that conversation before of course. :-)

[rsc]

Talked to @ianlancetaylor, @griesemer, and others at proposal review.

The current go line in the go.mod file means "use this exact version of the language semantics".

For a dependency module with a go.mod file with no go line, or no go.mod at all, we will assume the last version of go before we start making breaking changes. (Let's call that Go 1.L.) That way, imported old code keeps using the older version and does not break.

For the current module, once we start making breaking changes, if you run a go command in that module tree and there is no go line, the go command will add a go 1.X line where X is the current version of go. That way go.mod is "initialized" correctly and people don't have to go out of their way to get new language features.

Having decided for a given module which go version to use, the go command would pass -lang=xxx to the compiler when compiling a package. (For alternate build systems that don't yet know about -lang, we'd default to the compiler assuming the latest version of the language. But the go command would simply always pass -lang.) Note that this means all packages in one module get a consistent version of go, which might help for being sure it's OK to move code around inside one module.

For a dependency asking for Go 1.(N+1) or later, when the current go version is Go 1.N, one option is to refuse to build. Another option is to try compiling as Go 1.N and see if the build succeeds. If we never redefine the meaning of existing syntax, then it should be safe to conclude from a successful build that everything is OK. And if the build fails, then it can give the error and say 'by the way, this was supposed to use a newer version of Go, so maybe that's the problem.' @ianlancetaylor has been advocating for this behavior, which would make as much code build as possible, even if people accidentally set the go version line too new. It sounds like this should work - in part because we've all agreed not to redefine the meaning of any existing syntax - so we should probably do it and only roll it back if there is a major problem.

[rsc]

Leaving open for additional comments.

[gopherbot]

Change https://golang.org/cl/144340 mentions this issue: cmd/compile: add -lang flag to specify language version

[wsc1]

+1: reclassifying core std lib vs penumbra
+1: -lang compiler flag
-1: per file directives

I think more thought needs to be put into how to deal with mixed -lang= versions. Neither solution "refuse to build" or "issue a warning" seems right because: "refuse to build" means collecting packages/modules for a product could become quite involved and difficult for trivial reasons. "issue a warning" is against the go practice of failing to build unless we're sure it works. One option may be to explore introducing binary compatibility at the same time -lang= is introduced. So if -lang= is first available in go1.12, then from go1.12 onwards, a commitment to binary compatibility of packages/modules could be made. In this case, -lang=x in one module and lang=y in another with x!=y could run and a warning could be issued, but only if the caller requests uniformity of lang= or some other constraints on it.

question: lang meaning changes: #24543 changes the meaning of a loop without function calls to include the possibility of pre-emption by the Go scheduler/runtime. How does this proposal relate?

[wsc1]

I think it's important that the language version be a maximum required language version, not a minimum. Otherwise, everyone is forced up to the version of Go used by whoever built the newest library that they depend on.

This point also relates to language meaning changes. As long as the language meaning/semantics is not precise, it doesn't make much sense to declare the meaning of go 1.N compatible with go 1.N-1 (which is equivalent to specifying a minimum version). Case in point is #24543, which (while solving a myriad problems, also) changes the semantics to something less expressive and has restrictive implications for lower level systems programming. If the language semantics (memory model, scheduling, relation to system calls) were precise, then backwards compatibility (minimum version specification) would be reasonable. But it's just not the case.

[networkimprov]

@wsc1 maybe propose runtime.Get/SetPreempt() in a new issue, with milestone 1.12?

[ianlancetaylor]

@wsc1 There is no intention of supporting binary compatibility. That's not what the -lang option means. If you are building your program with Go 1.N, then you have to build everything with Go 1.N. We've only ever provided source-level compatibility, and that is all we ever expect to provide.

The -lang option doesn't mean "build with Go 1.X", it means "build with the language semantics of Go 1.X". Just like the -std option of GCC or clang.

What @rsc outlined above is: if the module requests a newer language version, try to build it with the current language version. If that works, fine. If it fails, report the error, and also say that it may be due to the fact that the module requires a newer language version. I think that is consistent with what you are suggesting.

[ianlancetaylor]

@wsc1 You are correct that source compatibility is not going to handle all possible cases. So it goes.

[wsc1]

@wsc1 maybe propose runtime.Get/SetPreempt() in a new issue, with milestone 1.12?

Already proposed (as something that would work providing it isn't implemented by means of a pre-empting sys call) in that issue.

Also #21314 I proposed a user compiler directive to replace go:nosplit.

No responses (except a thumbs down :), no idea where the pre-emption stuff is or whether it will be implemented, so timeline is awfully tight for figuring out a solution for 1.12, and I'm not a compiler or runtime expert so...

[wsc1]

The -lang option doesn't mean "build with Go 1.X", it means "build with the language semantics of Go 1.X". Just like the -std option of GCC or clang.

One potential difference: I think I can link -std=C11 with std=C99, at least in many cases, provided the exported .h includes both or an intersection.

[ianlancetaylor]

This is implemented.

[gopherbot]

Change https://golang.org/cl/151358 mentions this issue: [release-branch.go1.11] cmd/go: don't fail if requested Go version is later than current one

[tmornini]

Please don’t use any form of “language” to describe this.

The language is Go. We’re all clear on that, right?

How about grammar instead?

[ianlancetaylor]

@tmornini When we speak of Go 1.8 and Go 1.9, we are always speaking of the Go language, but at the same time there is a different between the Go 1.8 language and the Go 1.9 language, in that the latter accepts type aliases.

[tmornini]

@ianlancetaylor Yes?!?

I’m suggesting a small change from

-lang=go1.12

to:

-grammar=go1.12

Or, simpler:

-grammar=1.12

The point being: it’s all go...

[ianlancetaylor]

@tmornini Thanks, I think I understand your point, and I disagree. Go 1.8 and Go 1.9 are (slightly) different languages.

[gopherbot]

Change https://golang.org/cl/164878 mentions this issue: cmd/go: document GoVersion field in Module struct