testing: new release on 2021-10-18 (34.20211016.2.1)

[saqibali-2k]

First, verify that you meet all the prerequisites

Name this issue testing: new release on YYYY-MM-DD with today's date. Once the pipeline spits out the new version ID, you can append it to the title e.g. (31.20191117.2.0).

Pre-release

Promote testing-devel changes to testing

• Add the ok-to-promote label to the issue
• Review the promotion PR opened by the bot against the testing branch on https://github.com/coreos/fedora-coreos-config
• Once CI has passed, merge it

Build

• Start a pipeline build (select testing, leave all other defaults)
• Post a link to the jobs as a comment to this issue
  • x86_64
  • aarch64 (multi-arch pipeline)
• Wait for the job to finish and succeed
  • x86_64
  • aarch64

Sanity-check the build

Using the the build browser for the testing stream:

• Verify that the parent commit and version match the previous testing release (in the future, we'll want to integrate this check in the release job)
  • x86_64
  • aarch64
• Check kola AWS runs to make sure they didn't fail
  • x86_64
  • aarch64
• Check kola GCP run to make sure it didn't fail
• Check kola OpenStack run to make sure it didn't fail

⚠️ Release ⚠️

IMPORTANT: this is the point of no return here. Once the OSTree commit is
imported into the unified repo, any machine that manually runs rpm-ostree upgrade will have the new update.

Run the release job

• Run the release job, filling in for parameters testing and the new version ID
• Post a link to the job as a comment to this issue
• Wait for job to finish

At this point, Cincinnati will see the new release on its next refresh and create a corresponding node in the graph without edges pointing to it yet.

Refresh metadata (stream and updates)

From a checkout of this repo:

• Update stream metadata, by running:

fedora-coreos-stream-generator -releases=https://fcos-builds.s3.amazonaws.com/prod/streams/testing/releases.json  -output-file=streams/testing.json -pretty-print

• Add a rollout. For a 48-hour rollout starting at 10 AM ET, run:

./rollout.py add testing <version> "10 am ET" 48

• Commit the changes and open a PR against the repo. Paste the output of make print-rollouts into the PR description.
• Post a link to the PR as a comment to this issue
• Wait for the PR to be approved.
• Once approved, merge it and verify that the sync-stream-metadata job syncs the contents to S3
• Verify the new version shows up on the download page
• Verify the incoming edges are showing up in the update graph.
  • x86_64
  • aarch64

Update graph manual check

curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=x86_64&stream=testing&rollout_wariness=0'
curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=aarch64&stream=testing&rollout_wariness=0'

NOTE: In the future, most of these steps will be automated.

Housekeeping

• If one doesn't already exist, open an issue in this repo for the next release in this stream. Use the approximate date of the release in the title.
• Issues opened via the previous link will automatically create a linked Jira card. Assign the GitHub issue and Jira card to the next person in the rotation.

[sohankunkerkar]

Pipeline builds:

1. x86: https://jenkins-fedora-coreos.apps.ocp.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline/3045/
2. aarch64: https://jenkins-fedora-coreos.apps.ocp.ci.centos.org/job/multi-arch-pipeline/205/

[sohankunkerkar]

multi-arch pipeline job failed

[dustymabe]

multi-arch pipeline job failed

Yes it failed with what looks like an infra flake:

--- FAIL: ostree.unlock/discard (5031.58s)
[2021-10-19T21:40:09.369Z]             unlock.go:162: Failed to reboot machine: machine "1f8d6afb-e3d6-4653-ad26-2782593d0482" failed to start: ssh journalctl failed: failed to retrieve boot ID: : ssh: handshake failed: read tcp 127.0.0.1:47374->127.0.0.1:40259: read: connection reset by peer:

but the re-run succeeded:

[2021-10-19T21:40:09.369Z] ======== Re-running failed tests (flake detection) ========
...
[2021-10-19T21:41:43.487Z]     --- PASS: ostree.unlock/discard (36.47s)

I've started a new run: multi-arch-pipeline#207

[dustymabe]

I've started a new run: multi-arch-pipeline#207

Similar problem:

03:16:26  --- FAIL: coreos.ignition.mount.disks (5092.69s)
03:16:26          mount.go:137: could not reboot machine: machine "bd09c9cd-8120-4014-9ddb-7832b59b9492" failed to start: ssh journalctl failed: failed to retrieve boot ID: : ssh: handshake failed: read tcp 127.0.0.1:39830->127.0.0.1:42847: read: connection reset by peer:

but the re-run passed.

Trying again: multi-arch-pipeline#210

[sohankunkerkar]

Release job: https://jenkins-fedora-coreos.apps.ocp.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline-release/833/

[sohankunkerkar]

rollout PR: #389