Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/cmd/relui: tag golang.org/x repos continuously #48523

Closed
toothrot opened this issue Sep 21, 2021 · 41 comments
Closed

x/build/cmd/relui: tag golang.org/x repos continuously #48523

toothrot opened this issue Sep 21, 2021 · 41 comments
Assignees
@heschi
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Unreleased

Comments

@toothrot
Copy link
Contributor

Plan: once a month (or whatever time unit)
For each x repo in dependency order (lowest first)
go get update all the deps to their latest tags
Wait for that to be green
Tag it with next version

Can probably shoehorn into existing process, maybe not running automatically at first.
0.1.0, 0.2.0, 0.3.0...
@toothrot toothrot added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Sep 21, 2021
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Sep 21, 2021
@gopherbot gopherbot added this to the Unreleased milestone Sep 21, 2021
@toothrot toothrot added the Soon This needs action soon. (recent regressions, service outages, unusual time-sensitive situations) label Oct 5, 2021
@dmitshur dmitshur changed the title x/build/updatedstd: tag imported versions of repos x/build: tag imported versions of repos Nov 9, 2021
@dmitshur
Copy link
Contributor

dmitshur commented Feb 8, 2022 •
edited
Loading

Wait for that to be green

Automating this process will be blocked on coming up with a machine-understandable criteria of what "green" means.

@dmitshur dmitshur mentioned this issue Mar 16, 2022
Open
@dmitshur dmitshur mentioned this issue Aug 5, 2022
Closed
@heschi heschi removed the Soon This needs action soon. (recent regressions, service outages, unusual time-sensitive situations) label Aug 11, 2022
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/423036 mentions this issue: internal/workflow: clarify ownership

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/423037 mentions this issue: internal/workflow: add expansions

gopherbot pushed a commit to golang/build that referenced this issue Aug 11, 2022
@heschi @gopherbot
internal/workflow: clarify ownership
e825973 
For golang/go#48523.

Change-Id: Ie55cdf3c90687d065fb1b81d0a9f97a81d6ffd1f
Reviewed-on: https://go-review.googlesource.com/c/build/+/423036
Reviewed-by: Jenny Rakoczy <[email protected]>
Auto-Submit: Heschi Kreinick <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
@heschi heschi self-assigned this Aug 12, 2022
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/425088 mentions this issue: internal/task: start x repo tagging tasks

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/425089 mentions this issue: internal/task: continue x repo tagging

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/425090 mentions this issue: internal/relui,internal/task: move useful structs to task

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/425091 mentions this issue: internal/task: finish initial implementation of x/ repo tagging

gopherbot pushed a commit to golang/build that referenced this issue Aug 25, 2022
@heschi
internal/workflow: add expansions
bf01fcc 
One of the upcoming use cases for relui, tagging x/ repos, has a
dependency graph we can't know without doing significant work. This is
in conflict with one of the workflow package's design principles, that
workflows be defined ahead of time. So here's one idea for bending that
rule.

Introduce the concept of an expansion task. An expansion task runs the
same as a normal task, but in addition to its inputs it receives (a copy
of) the workflow definition and can add tasks and outputs to it as
usual. Once the task finishes running, the workflow swaps in the new
definition and continues. Other tasks can run concurrently.

One significant difference is that the expansion task must run every
time the workflow is started or resumed, so they really should be pure
functions with no network access or side effects. To encourage this,
they don't receive a Context.

Introducing this required some refactoring of the existing Run loop; we
need to allow these apparently-unused tasks to run, and deal with new
tasks popping up mid-run. But overall I think it's not too huge a
distortion to the design.

For golang/go#48523.

Change-Id: I92c164883e06474fa951abba5fa18fd78eacb8ce
Reviewed-on: https://go-review.googlesource.com/c/build/+/423037
Run-TryBot: Heschi Kreinick <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Reviewed-by: Jenny Rakoczy <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/425495 mentions this issue: internal/task: integration test for x repo tagging

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. labels Sep 14, 2022
gopherbot pushed a commit to golang/build that referenced this issue Sep 15, 2022
@heschi @gopherbot
internal/task: start x repo tagging tasks
e7573ff 
First step: list all our projects/repositories, and select those that
have a go.mod labeling them as golang.org/x as candidates to release as
the next version, or v0.1.0 for untagged repositories.

For golang/go#48523.

Change-Id: Ice92319a0726daf3bf5f94581582d8802640dffc
Reviewed-on: https://go-review.googlesource.com/c/build/+/425088
Reviewed-by: Dmitri Shuralyov <[email protected]>
Auto-Submit: Heschi Kreinick <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
gopherbot pushed a commit to golang/build that referenced this issue Sep 15, 2022
@heschi @gopherbot
internal/task: continue x repo tagging
58cf979 
Next step: expand the workflow with tagging tasks, currently all no-ops.

For golang/go#48523.

Change-Id: I48cb8804dc1db6c17b748b54a194e9ef64ef6bce
Reviewed-on: https://go-review.googlesource.com/c/build/+/425089
Reviewed-by: Dmitri Shuralyov <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Auto-Submit: Heschi Kreinick <[email protected]>
gopherbot pushed a commit to golang/build that referenced this issue Sep 15, 2022
@heschi @gopherbot
internal/relui,internal/task: move useful structs to task
5c85681 
For whatever reason I've decided to write the tagging workflow in the
task package, so I need the test helpers here. I also need the website
structs to find the latest Go release later.

For golang/go#48523.

Change-Id: I92a1430bd27f0aa90c9fccefa12fb057e7e7f864
Reviewed-on: https://go-review.googlesource.com/c/build/+/425090
Reviewed-by: Dmitri Shuralyov <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Auto-Submit: Heschi Kreinick <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/431356 mentions this issue: maintner/maintnerd/maintapi: validate page size in mixed mode

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/431355 mentions this issue: cmd/coordinator/internal/legacydash: uncap maintner request

gopherbot pushed a commit to golang/build that referenced this issue Sep 16, 2022
@heschi @gopherbot
cmd/coordinator/internal/legacydash: uncap maintner request
d7fb432 
We've been seeing the coordinator fail to schedule subrepo runs on
release branches. As far as I can tell, the problem is the request to
maintner: branch = "mixed" means to get at least one commit from each
branch in the repository, but we're asking for 30 commits when we have
50-some branches. So obviously some are going to be left out. That
absence breaks findWork when it tries to map commits to branches.

I don't fully understand which branches happen to be included and which
don't, but this behavior seems clearly wrong. Let maintner decide how
many responses to send back, currently hardcoded at 500.

For golang/go#48523.

Change-Id: I1792e89de96e83ceed6a430a7456b119f633869b
Reviewed-on: https://go-review.googlesource.com/c/build/+/431355
Reviewed-by: Dmitri Shuralyov <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
Auto-Submit: Heschi Kreinick <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
gopherbot pushed a commit to golang/build that referenced this issue Sep 20, 2022
@heschi @gopherbot
internal/task: finish initial implementation of x/ repo tagging
1bb28a5 
This one got away from me a little, sorry.

Add cycle detection and breaking: we need to break some cycles involving
x/tools. We could consider fixing them by creating nested modules but I
don't want to deal with that now.

Implement UpdateGoMod: fetch the latest Go release, download it to a
buildlet with network access, and run go get to update to the previously
tagged versions. But only do this if there are dependencies, both to
save time and because go get throws an error if you try to get nothing.

Implement Tag: if there hasn't been a change since the last tag, reuse
that one. Otherwise, tag it with the selected new verison.

For golang/go#48523.

Change-Id: I29c05eb639fd748af1e0e356596adc8c3f15ccf0
Reviewed-on: https://go-review.googlesource.com/c/build/+/425091
Run-TryBot: Heschi Kreinick <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
Auto-Submit: Heschi Kreinick <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
gopherbot pushed a commit to golang/build that referenced this issue Oct 19, 2022
@heschi @gopherbot
internal/task: remove initial tagging support
bc09ea8 
With the first run done this code is now dead. We can create manual tags
for future bootstrapping.

For golang/go#48523.

Change-Id: Iff147d095205f3e3687136d85eaaa08bae207069
Reviewed-on: https://go-review.googlesource.com/c/build/+/444117
Reviewed-by: Dmitri Shuralyov <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
Auto-Submit: Heschi Kreinick <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/443857 mentions this issue: internal/task: tidy gopls while tagging tools

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/444295 mentions this issue: go.mod: mark as requiring -compat 1.13

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/444156 mentions this issue: cmd/go/internal/modload: update TestQueryImport to pass with tagged versions of x/net

gopherbot pushed a commit that referenced this issue Oct 20, 2022
@gopherbot
cmd/go/internal/modload: update TestQueryImport to pass with tagged v…
3e6ca3a 
…ersions of x/net

For #48523.

Change-Id: Ied35d15462cbae1002e1db1e6e119a6c9f8323da
Reviewed-on: https://go-review.googlesource.com/c/go/+/444156
Run-TryBot: Bryan Mills <[email protected]>
Reviewed-by: Than McIntosh <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Auto-Submit: Bryan Mills <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/444435 mentions this issue: [release-branch.go1.19] cmd/go/internal/modload: update TestQueryImport to pass with tagged versions of x/net

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/444436 mentions this issue: [release-branch.go1.18] cmd/go/internal/modload: update TestQueryImport to pass with tagged versions of x/net

gopherbot pushed a commit that referenced this issue Oct 20, 2022
@heschi
[release-branch.go1.19] cmd/go/internal/modload: update TestQueryImpo…
8d10cc0 
…rt to pass with tagged versions of x/net

For #48523.

Change-Id: Ied35d15462cbae1002e1db1e6e119a6c9f8323da
Reviewed-on: https://go-review.googlesource.com/c/go/+/444156
Run-TryBot: Bryan Mills <[email protected]>
Reviewed-by: Than McIntosh <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Auto-Submit: Bryan Mills <[email protected]>
(cherry picked from commit 3e6ca3a)
Reviewed-on: https://go-review.googlesource.com/c/go/+/444435
Reviewed-by: Heschi Kreinick <[email protected]>
gopherbot pushed a commit that referenced this issue Oct 20, 2022
@heschi
[release-branch.go1.18] cmd/go/internal/modload: update TestQueryImpo…
828be9a 
…rt to pass with tagged versions of x/net

For #48523.

Change-Id: Ied35d15462cbae1002e1db1e6e119a6c9f8323da
Reviewed-on: https://go-review.googlesource.com/c/go/+/444156
Run-TryBot: Bryan Mills <[email protected]>
Reviewed-by: Than McIntosh <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Auto-Submit: Bryan Mills <[email protected]>
(cherry picked from commit 3e6ca3a)
Reviewed-on: https://go-review.googlesource.com/c/go/+/444436
Reviewed-by: Heschi Kreinick <[email protected]>
gopherbot pushed a commit to golang/tools that referenced this issue Oct 20, 2022
@heschi @gopherbot
go.mod: mark as requiring -compat 1.16
649df2e 
When the tagging process runs go mod tidy it needs to pass -compat 1.16.
This magic comment will cause it to do so. See CL 443857.

For golang/go#48523.

Change-Id: I6824a2c78462604b9d4737444b4d16d4c21ecc99
Reviewed-on: https://go-review.googlesource.com/c/tools/+/444295
Auto-Submit: Heschi Kreinick <[email protected]>
Reviewed-by: Robert Findley <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
gopherbot pushed a commit to golang/build that referenced this issue Oct 20, 2022
@heschi @gopherbot
internal/task: clean up tagging commit selection logic
4237ab0 
Clean up after the change to read the branch head later. Also fix a bug:
in the version tasks, we want to know exactly what commit we're tagging,
so we pass in the expected commit, and AwaitCL returns that if no CL was
necessary. But here we want to tag the latest branch head if there's no
CL necessary. So we need a different Await function.

For golang/go#48523.

Change-Id: I97f92b22d4f29edeb52d06c311fddaa6596da0c2
Reviewed-on: https://go-review.googlesource.com/c/build/+/444116
Reviewed-by: Dmitri Shuralyov <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
Auto-Submit: Heschi Kreinick <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
@dmitshur dmitshur changed the title x/build: tag imported versions of repos x/build/cmd/relui: tag golang.org/x repos continuously Oct 20, 2022
gopherbot pushed a commit to golang/build that referenced this issue Oct 20, 2022
@heschi @gopherbot
internal/task: tidy gopls while tagging tools
8be4277 
gopls has a replace directive that uses the local tools version, which
means that they have to be updated together.

This is currently a one-off, so rather than trying to figure it out
automatically tools is hardcoded. I'm not proud of it. But at least
there's a test! (?)

For golang/go#48523.

Change-Id: If59e669ba3ede08967b72ba3bb36b2c66e1a03a7
Reviewed-on: https://go-review.googlesource.com/c/build/+/443857
TryBot-Result: Gopher Robot <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
Auto-Submit: Heschi Kreinick <[email protected]>
@heschi
Copy link
Contributor

heschi commented Oct 20, 2022

I'm gonna call this done. We haven't run it fully end-to-end yet because of the hiccups with x/tools, but it got very close. One possible followup is to have it mail CLs to specified reviewers, since it turns out almost none of our repositories have owners any more, but I'll decide about that next time we run it.

@heschi heschi closed this as completed Oct 20, 2022
Repository owner moved this from In Progress to Done in Go Release Oct 20, 2022
@hyangah hyangah mentioned this issue Oct 21, 2022
Closed
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/445955 mentions this issue: internal/task: add support for tagging a single repository

gopherbot pushed a commit to golang/build that referenced this issue Oct 31, 2022
@heschi @gopherbot
internal/task: add support for tagging a single repository
b041a95 
To make one-off tagging less error-prone, run a miniature version of the
overall workflow to tag a specific repository.

Also includes a tiny bug fix for expansions with parameters.

For golang/go#48523.

Change-Id: Ic9089966aa28d20c716169b41121e2da21faf54a
Reviewed-on: https://go-review.googlesource.com/c/build/+/445955
Auto-Submit: Heschi Kreinick <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Run-TryBot: Heschi Kreinick <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
@bcmills bcmills mentioned this issue Nov 2, 2022
Closed
romaindoumenc pushed a commit to TroutSoftware/go that referenced this issue Nov 3, 2022
@romaindoumenc
cmd/go/internal/modload: update TestQueryImport to pass with tagged v…
732252c 
…ersions of x/net

For golang#48523.

Change-Id: Ied35d15462cbae1002e1db1e6e119a6c9f8323da
Reviewed-on: https://go-review.googlesource.com/c/go/+/444156
Run-TryBot: Bryan Mills <[email protected]>
Reviewed-by: Than McIntosh <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Auto-Submit: Bryan Mills <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/449038 mentions this issue: internal/task: final touches on x/ repo tagging

gopherbot pushed a commit to golang/build that referenced this issue Nov 9, 2022
@heschi @gopherbot
internal/task: final touches on x/ repo tagging
3029cba 
Tagging has now run to completion and I'm comfortable turning it loose.
Remove the approval requirement and send CLs to specified reviewers so
that it can be run as a scheduled workflow.

For golang/go#48523.

Change-Id: I2411a2680b09f877b8f4e5905ed51ae2e9ef2bdd
Reviewed-on: https://go-review.googlesource.com/c/build/+/449038
Run-TryBot: Heschi Kreinick <[email protected]>
Reviewed-by: Jenny Rakoczy <[email protected]>
Auto-Submit: Heschi Kreinick <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
andrew-d pushed a commit to tailscale/go that referenced this issue Dec 7, 2022
@andrew-d
[release-branch.go1.19] cmd/go/internal/modload: update TestQueryImpo…
862858c 
…rt to pass with tagged versions of x/net

For golang#48523.

Change-Id: Ied35d15462cbae1002e1db1e6e119a6c9f8323da
Reviewed-on: https://go-review.googlesource.com/c/go/+/444156
Run-TryBot: Bryan Mills <[email protected]>
Reviewed-by: Than McIntosh <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Auto-Submit: Bryan Mills <[email protected]>
(cherry picked from commit 3e6ca3a)
Reviewed-on: https://go-review.googlesource.com/c/go/+/444435
Reviewed-by: Heschi Kreinick <[email protected]>
@thaJeztah thaJeztah mentioned this issue Dec 7, 2022
Merged
@thaJeztah thaJeztah mentioned this issue Jan 14, 2023
Open
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/501835 mentions this issue: go.mod: delete repeated "indirect"

gopherbot pushed a commit to golang/text that referenced this issue Jun 8, 2023
@dmitshur @gopherbot
go.mod: delete repeated "indirect"
d61dd50 
It got added by 'go mod tidy' in CL 471399
because the '; ' separator was missing.

For golang/go#48523.
For golang/go#58737.

Change-Id: I63642a4f49beb9679f9c5bbb4e2aa8ed0cc73c9e
Reviewed-on: https://go-review.googlesource.com/c/text/+/501835
TryBot-Result: Gopher Robot <[email protected]>
Auto-Submit: Dmitri Shuralyov <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
Reviewed-by: Ian Lance Taylor <[email protected]>
Run-TryBot: Dmitri Shuralyov <[email protected]>
@golang golang locked and limited conversation to collaborators Jun 7, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@heschi heschi

Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
Go Release
Archived in project
Milestone
Unreleased
Development

No branches or pull requests
5 participants
@toothrot @FiloSottile @dmitshur @gopherbot @heschi