Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

stable: new release 34.20210919.3.0 on 2021-10-04 #373

Closed
33 tasks done
lucab opened this issue Sep 21, 2021 · 6 comments
Closed
33 tasks done

stable: new release 34.20210919.3.0 on 2021-10-04 #373

lucab opened this issue Sep 21, 2021 · 6 comments
Assignees
@ravanelli @travier
Labels
jira kind/release ok-to-promote

Comments

@lucab
Copy link
Contributor

lucab commented Sep 21, 2021 •
edited by ravanelli
Loading

First, verify that you meet all the prerequisites

Name this issue stable: new release on YYYY-MM-DD with today's date. Once the pipeline spits out the new version ID, you can append it to the title e.g. (31.20191117.3.0).

Pre-release

Promote testing changes to stable

Build

  • Start a pipeline build (select stable, leave all other defaults)
  • Post a link to the job as a comment to this issue
  • Wait for the job to finish and succeed
    • x86_64
    • aarch64

Sanity-check the build

Using the the build browser for the stable stream:

  • Verify that the parent commit and version match the previous stable release (in the future, we'll want to integrate this check in the release job)
    • x86_64
    • aarch64
  • Check kola AWS runs to make sure they didn't fail
    • x86_64
    • aarch64
  • Check kola GCP run to make sure it didn't fail
  • Check kola OpenStack run to make sure it didn't fail

⚠️ Release ⚠️

IMPORTANT: this is the point of no return here. Once the OSTree commit is
imported into the unified repo, any machine that manually runs rpm-ostree upgrade will have the new update.

Run the release job

  • Run the release job, filling in for parameters stable and the new version ID
  • Post a link to the job as a comment to this issue
  • Wait for job to finish

At this point, Cincinnati will see the new release on its next refresh and create a corresponding node in the graph without edges pointing to it yet.

Refresh metadata (stream and updates)

From a checkout of this repo:

  • Update stream metadata, by running:
fedora-coreos-stream-generator -releases=https://fcos-builds.s3.amazonaws.com/prod/streams/stable/releases.json  -output-file=streams/stable.json -pretty-print
  • Add a rollout. For a 48-hour rollout starting at 10 AM ET, run:
./rollout.py add stable <version> "10 am ET" 48
  • Commit the changes and open a PR against the repo. Paste the output of make print-rollouts into the PR description.
  • Post a link to the PR as a comment to this issue
  • Wait for the PR to be approved.
  • Once approved, merge it and verify that the sync-stream-metadata job syncs the contents to S3
  • Verify the new version shows up on the download page
  • Verify the incoming edges are showing up in the update graph.
Update graph manual check 
curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=x86_64&stream=stable&rollout_wariness=0'
curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=aarch64&stream=stable&rollout_wariness=0'

NOTE: In the future, most of these steps will be automated.

Housekeeping

  • If one doesn't already exist, open an issue in this repo with the approximate date in the title of the next release in this stream.
  • Issues opened via the previous link will automatically create a linked Jira card. Assign the GitHub issue and Jira card to the next person in the rotation.
@lucab lucab mentioned this issue Sep 21, 2021
Closed
36 tasks
@dustymabe dustymabe assigned travier and ravanelli and unassigned sohankunkerkar Sep 21, 2021
@travier travier mentioned this issue Oct 4, 2021
Merged
@coreosbot-releng coreosbot-releng mentioned this issue Oct 4, 2021
Merged
@ravanelli
Copy link
Member

ravanelli commented Oct 4, 2021 •
edited
Loading

Pipeline builds:

@ravanelli ravanelli changed the title stable: new release on 2021-10-04 stable: new release 34.20210919.3.0 on 2021-10-04 Oct 5, 2021
@ravanelli
Copy link
Member

The kola AWS run for aarch64 failed 2 tests with a known issue describe in coreos/fedora-coreos-tracker#803

Looking at the logs both tests had the same behavior as described in the issue above.

[   13.312601] multipathd[571]: --------shut down-------
[   13.313451] multipathd[914]: 13.301266 | /etc/multipath.conf does not exist, blacklisting all devices.
[   13.314717] multipathd[914]: 13.301300 | You can run "/sbin/mpathconf --enable" to create
         Starting [0;1;39mdracut pre-pivot and cleanup hook [0m[   13.316290] multipathd[914]: 13.301303 | /etc/multipath.conf. See man mpathconf(8) for more details
...[   13.317737] systemd[1]: Reached target Initrd File Systems.

[   13.318672] systemd[1]: Reached target Initrd Default Target.
[   13.319577] systemd[1]: Condition check resulted in dracut mount hook being skipped.
[   13.320696] systemd[1]: Starting dracut pre-pivot and cleanup hook...
[   13.378024] multipathd[914]: error -5 receiving packet
[   13.378839] systemd[1]: multipathd.service: Control process exited, code=exited, status=1/FAILURE
[   13.400461] dracut-pre-pivot[936]: 13.392669 | /etc/multipath.conf does not exist, blacklisting all devices.
[   13.394190] dracut-pre-pivot[936]: 13.392669 | /etc/multipath.conf does not exist, blacklisti[   13.402578] dracut-pre-pivot[936]: 13.392703 | You can run "/sbin/mpathconf --enable" to create
ng all devices.
[[0;32m  OK  [0m] Stopped [0;1;39mDevice-Mapper Multipath Device Controller[[   13.404498] dracut-pre-pivot[936]: 13.392707 | /etc/multipath.conf. See man mpathconf(8) for more details
0m.
[   13.396268] dracut-pre-pivot[936]: 13.392703 | You can run "/sbin/mpathconf --enable" to create
[   13.398520] dracut-pre-pivot[936]: 13.392707 | /etc/multipath.conf. See man mpathconf(8) for more details
[   13.408281] systemd[1]: multipathd.service: Failed with result 'exit-code'.

Tests with the error:

--- FAIL: coreos.ignition.journald-log (384.62s)
14:47:41          harness.go:1130: Cluster failed starting machines: machine "i-0ed7481415fa06aee" failed basic checks: some systemd units failed:
14:47:41  multipathd.service

15:10:18  --- FAIL: non-exclusive-tests (555.53s)
15:10:18          harness.go:1130: Cluster failed starting machines: machine "i-01e10a095d220258f" failed basic checks: some systemd units failed:
15:10:18  multipathd.service

@ravanelli
Copy link
Member

The kola OpenStack run are taking too long for instances to come up. We double checked that we are good to go, at this point the tests went well.

19:24:19  --- FAIL: ostree.remote (614.58s)
19:24:19          harness.go:1130: Cluster failed starting machines: waiting for instance to run: time limit exceeded

19:34:37  --- FAIL: coreos.selinux.enforce (610.86s)
19:34:37          harness.go:1130: Cluster failed starting machines: waiting for instance to run: time limit exceeded

@ravanelli
Copy link
Member

Release job:

@ravanelli
Copy link
Member

PR created:

@ravanelli
Copy link
Member

Next issue: #384

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ravanelli ravanelli

@travier travier

Labels
jira kind/release ok-to-promote
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@lucab @ravanelli @sohankunkerkar @saqibali-2k @travier