s390x: add secure-execution support #1353

nikita-dubrovskii · 2021-11-29T14:34:00Z

This adds genprotimg tool, for creating encrypted sd-boot image, and turns on (if required) SecureExecution during firstboot-complete
Depends on: coreos/coreos-installer#851

bgilbert

As previously agreed, hold until feature planning is complete.

No longer blocked on planning.

travier

Only added some suggestions as I don't have the context to review.

overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-growfs.sh

overlay.d/05core/usr/lib/dracut/modules.d/35coreos-ignition/coreos-gpt-setup.sh

overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-growfs.sh

jlebon · 2022-05-24T14:22:47Z

overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-growfs.sh

+                device=$(realpath /dev/disk/by-label/"${DM_NAME}")
+                eval $(udevadm info --query=property --export $device | grep PARTN=)
+                eval $(lsblk -o PKNAME --nodeps --noheadings --paths --pairs $device)
+                parted $PKNAME resizepart $PARTN 100%


Isn't this already done by growpart at the earlier part iteration?

if it does, than it's broken or doesn't work wit LUKS pre-encrypted partitions

That growpart is tested by CI at least. There's actually a LUKS test which does this: https://github.com/coreos/fedora-coreos-config/tree/testing-devel/tests/kola/root-reprovision/luks. That should be the same setup as we have here (root on LUKS on partition).

Some debug output:

[ 19.502720] ignition-ostree-growfs[1292]: ++ realpath /dev/disk/by-label/root [ 19.503058] ignition-ostree-growfs[1288]: + partition=/dev/dm-0 [ 19.503216] ignition-ostree-growfs[1293]: + lsblk -no TYPE /dev/dm-0 ... [ 19.504843] ignition-ostree-growfs[1288]: + current_blkdev=/dev/dm-0 [ 19.504855] ignition-ostree-growfs[1288]: + true [ 19.504933] ignition-ostree-growfs[1296]: ++ lsblk --paths --nodeps --pairs -o NAME,TYPE,PKNAME /dev/dm-0 [ 19.505523] ignition-ostree-growfs[1288]: + eval 'NAME="/dev/mapper/crypt_rootfs" TYPE="crypt" PKNAME=""' [ 19.505538] ignition-ostree-growfs[1288]: ++ NAME=/dev/mapper/crypt_rootfs [ 19.505550] ignition-ostree-growfs[1288]: ++ TYPE=crypt [ 19.505561] ignition-ostree-growfs[1288]: ++ PKNAME= [ 19.505892] ignition-ostree-growfs[1298]: +++ lsblk -dno MAJ:MIN /dev/mapper/crypt_rootfs

So we always enter crypt case-switch first. And part is unreachable in that case

That should be the same setup as we have here (root on LUKS on partition).

pls correct me if i'm wrong, but that test creates LUKSed root with ignition.disks stage

Yes indeed, that test uses Ignition to create the LUKS root. I thought the end result would be the same, but maybe I'm missing something.

What's the output of lsblk on a booted s390x system with your cosa patch? Maybe we should do a screen sharing session for this.

[core@localhost ~]$ lsblk -f NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS vda |-vda1 ext4 1.0 se 71589f1a-5698-4d22-8877-91317173ec61 |-vda3 crypto_LUKS 2 crypt_bootfs 8dc99436-f245-470e-81c3-d2f07d49dcfa | `-crypt_bootfs ext4 1.0 boot a774674a-e0d6-11ec-8ab8-0f20fe3c51e1 224M 26% /boot `-vda4 crypto_LUKS 2 crypt_rootfs 5e9d4812-87e5-4ed7-8c68-dd1874031839 `-crypt_rootfs xfs root a8ec61ea-e0d6-11ec-a07c-525400123456 7.9G 16% /var /sysroot/ostree/deploy/fedora-coreos/var /usr /etc / /sysroot

OK right, looking at this more, I think we need to adapt the top of the file to handle Secure Execution correctly. Can you try the following patch on top of this PR?

diff --git a/manifests/bootable-rpm-ostree.yaml b/manifests/bootable-rpm-ostree.yaml index b0659834..8c41320c 100644 --- a/manifests/bootable-rpm-ostree.yaml +++ b/manifests/bootable-rpm-ostree.yaml @@ -29,8 +29,6 @@ packages-s390x: # provided by s390utils-base, but soon will be -core too. - /usr/sbin/zipl - /usr/bin/genprotimg - # resize LUKS encrypted rootfs - - /usr/sbin/parted packages-x86_64: - grub2 grub2-efi-x64 efibootmgr shim - microcode_ctl diff --git a/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-growfs.sh b/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-growfs.sh index 1a0d68a7..946de6e1 100755 --- a/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-growfs.sh +++ b/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-growfs.sh @@ -17,8 +17,21 @@ path=/sysroot # this shouldn't happen for us but we're being conservative. src=$(findmnt -nvr -o SOURCE "$path" | tail -n1) +# IBM SecureExecution +secure_execution=0 +if [[ $(uname -m) == s390x ]] && [[ -e /sys/firmware/uv/prot_virt_guest ]]; then + secure_execution=$(cat /sys/firmware/uv/prot_virt_guest) +fi + if [ ! -f "${saved_partstate}" ]; then partition=$(realpath /dev/disk/by-label/root) + # in the Secure Execution case, the rootfs is pre-baked on LUKS, so + # `partition` is actually the device mapper device; get its parent + if [[ "${secure_execution}" != "0" ]]; then + # lsblk doesn't print PKNAME of crypt devices with --nodeps + MAJMIN=$(echo $(lsblk -dno MAJ:MIN "${partition}")) + partition=/dev/$(ls "/sys/dev/block/${MAJMIN}/slaves") + fi else # The rootfs was reprovisioned. Our rule in this case is: we only grow if # the partition backing the rootfs is the same and its size didn't change @@ -40,12 +53,6 @@ else fi fi -# IBM SecureExecution -secure_execution=0 -if [[ $(uname -m) == s390x ]] && [[ -e /sys/firmware/uv/prot_virt_guest ]]; then - secure_execution=$(cat /sys/firmware/uv/prot_virt_guest) -fi - # Go through each blockdev in the hierarchy and verify we know how to grow them lsblk -no TYPE "${partition}" | while read dev; do case "${dev}" in @@ -86,22 +93,13 @@ while true; do fi ;; crypt) - eval $(udevadm info --query=property --export "${NAME}") - if [[ "${secure_execution}" == "0" ]]; then - # XXX: yuck... we need to expose this sanely in clevis - (. /usr/bin/clevis-luks-common-functions - # lsblk doesn't print PKNAME of crypt devices with --nodeps - PKNAME=/dev/$(ls "/sys/dev/block/${MAJMIN}/slaves") - clevis_luks_unlock_device "${PKNAME}" | cryptsetup resize -d- "${DM_NAME}" - ) - else - device=$(realpath /dev/disk/by-label/"${DM_NAME}") - eval $(udevadm info --query=property --export $device | grep PARTN=) - eval $(lsblk -o PKNAME --nodeps --noheadings --paths --pairs $device) - growpart "${PKNAME}" "${PARTN}" || : - parted $PKNAME resizepart $PARTN 100% - cryptsetup resize --key-file=/etc/luks/root "${DM_NAME}" - fi + # XXX: yuck... we need to expose this sanely in clevis + (. /usr/bin/clevis-luks-common-functions + eval $(udevadm info --query=property --export "${NAME}") + # lsblk doesn't print PKNAME of crypt devices with --nodeps + PKNAME=/dev/$(ls "/sys/dev/block/${MAJMIN}/slaves") + clevis_luks_unlock_device "${PKNAME}" | cryptsetup resize -d- "${DM_NAME}" + ) ;; # already checked *) echo "unreachable" 1>&2; exit 1 ;; diff --git a/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/module-setup.sh b/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/module-setup.sh index 1ce53c48..bf9a7872 100755 --- a/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/module-setup.sh +++ b/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/module-setup.sh @@ -46,7 +46,6 @@ install() { xfs_admin \ xfs_growfs \ wc \ - parted \ wipefs # growpart deps

OK right, looking at this more, I think we need to adapt the top of the file to handle Secure Execution correctly. Can you try the following patch on top of this PR?

yup, thx. only had to skip clevis part in SE. not sure, but maybe clevis bind could be used to automate the unlocking here. will check that

looks like we cannot and shouldn't use clevis

overlay.d/05core/usr/lib/dracut/modules.d/35coreos-ignition/coreos-gpt-setup.sh

jlebon · 2022-05-30T19:07:14Z

overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-growfs.sh

+                device=$(realpath /dev/disk/by-label/"${DM_NAME}")
+                eval $(udevadm info --query=property --export $device | grep PARTN=)
+                eval $(lsblk -o PKNAME --nodeps --noheadings --paths --pairs $device)
+                parted $PKNAME resizepart $PARTN 100%


Yes indeed, that test uses Ignition to create the LUKS root. I thought the end result would be the same, but maybe I'm missing something.

What's the output of lsblk on a booted s390x system with your cosa patch? Maybe we should do a screen sharing session for this.

Signed-off-by: Nikita Dubrovskii <[email protected]>

jlebon · 2022-06-24T16:51:15Z

manifests/bootable-rpm-ostree.yaml

@@ -28,6 +28,9 @@ packages-s390x:
  # On Fedora, this is provided by s390utils-core. on RHEL, this is for now
  # provided by s390utils-base, but soon will be -core too.
  - /usr/sbin/zipl
+  - /usr/bin/genprotimg


Are you able to compose an s390x FCOS with this? I suspect it will fail because of:

fedora-coreos-config/manifests/fedora-coreos.yaml

Line 114 in 401c56b

- perl

We'll have to temporarily split that out into a separately conditional-included treefile until coreos/fedora-coreos-tracker#1217 is fixed.

Are you able to compose an s390x FCOS with this?

yes, no issues at all (metal/metal4k/qemu/secex/live/dasd images)

Ahh interesting. So actually, /usr/bin/perl is provided by perl-interpreter, which is why this works.

I'll add it to the list, but will take care of doing the include splitting to not break s390x builds.

Follow-up to this in #1848 now that we've moved it out of FCOS again.

jlebon · 2022-06-24T17:17:23Z

overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-growfs.sh

+                device=$(realpath /dev/disk/by-label/"${DM_NAME}")
+                eval $(udevadm info --query=property --export $device | grep PARTN=)
+                eval $(lsblk -o PKNAME --nodeps --noheadings --paths --pairs $device)
+                parted $PKNAME resizepart $PARTN 100%


OK right, looking at this more, I think we need to adapt the top of the file to handle Secure Execution correctly. Can you try the following patch on top of this PR?

diff --git a/manifests/bootable-rpm-ostree.yaml b/manifests/bootable-rpm-ostree.yaml index b0659834..8c41320c 100644 --- a/manifests/bootable-rpm-ostree.yaml +++ b/manifests/bootable-rpm-ostree.yaml @@ -29,8 +29,6 @@ packages-s390x: # provided by s390utils-base, but soon will be -core too. - /usr/sbin/zipl - /usr/bin/genprotimg - # resize LUKS encrypted rootfs - - /usr/sbin/parted packages-x86_64: - grub2 grub2-efi-x64 efibootmgr shim - microcode_ctl diff --git a/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-growfs.sh b/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-growfs.sh index 1a0d68a7..946de6e1 100755 --- a/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-growfs.sh +++ b/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-growfs.sh @@ -17,8 +17,21 @@ path=/sysroot # this shouldn't happen for us but we're being conservative. src=$(findmnt -nvr -o SOURCE "$path" | tail -n1) +# IBM SecureExecution +secure_execution=0 +if [[ $(uname -m) == s390x ]] && [[ -e /sys/firmware/uv/prot_virt_guest ]]; then + secure_execution=$(cat /sys/firmware/uv/prot_virt_guest) +fi + if [ ! -f "${saved_partstate}" ]; then partition=$(realpath /dev/disk/by-label/root) + # in the Secure Execution case, the rootfs is pre-baked on LUKS, so + # `partition` is actually the device mapper device; get its parent + if [[ "${secure_execution}" != "0" ]]; then + # lsblk doesn't print PKNAME of crypt devices with --nodeps + MAJMIN=$(echo $(lsblk -dno MAJ:MIN "${partition}")) + partition=/dev/$(ls "/sys/dev/block/${MAJMIN}/slaves") + fi else # The rootfs was reprovisioned. Our rule in this case is: we only grow if # the partition backing the rootfs is the same and its size didn't change @@ -40,12 +53,6 @@ else fi fi -# IBM SecureExecution -secure_execution=0 -if [[ $(uname -m) == s390x ]] && [[ -e /sys/firmware/uv/prot_virt_guest ]]; then - secure_execution=$(cat /sys/firmware/uv/prot_virt_guest) -fi - # Go through each blockdev in the hierarchy and verify we know how to grow them lsblk -no TYPE "${partition}" | while read dev; do case "${dev}" in @@ -86,22 +93,13 @@ while true; do fi ;; crypt) - eval $(udevadm info --query=property --export "${NAME}") - if [[ "${secure_execution}" == "0" ]]; then - # XXX: yuck... we need to expose this sanely in clevis - (. /usr/bin/clevis-luks-common-functions - # lsblk doesn't print PKNAME of crypt devices with --nodeps - PKNAME=/dev/$(ls "/sys/dev/block/${MAJMIN}/slaves") - clevis_luks_unlock_device "${PKNAME}" | cryptsetup resize -d- "${DM_NAME}" - ) - else - device=$(realpath /dev/disk/by-label/"${DM_NAME}") - eval $(udevadm info --query=property --export $device | grep PARTN=) - eval $(lsblk -o PKNAME --nodeps --noheadings --paths --pairs $device) - growpart "${PKNAME}" "${PARTN}" || : - parted $PKNAME resizepart $PARTN 100% - cryptsetup resize --key-file=/etc/luks/root "${DM_NAME}" - fi + # XXX: yuck... we need to expose this sanely in clevis + (. /usr/bin/clevis-luks-common-functions + eval $(udevadm info --query=property --export "${NAME}") + # lsblk doesn't print PKNAME of crypt devices with --nodeps + PKNAME=/dev/$(ls "/sys/dev/block/${MAJMIN}/slaves") + clevis_luks_unlock_device "${PKNAME}" | cryptsetup resize -d- "${DM_NAME}" + ) ;; # already checked *) echo "unreachable" 1>&2; exit 1 ;; diff --git a/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/module-setup.sh b/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/module-setup.sh index 1ce53c48..bf9a7872 100755 --- a/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/module-setup.sh +++ b/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/module-setup.sh @@ -46,7 +46,6 @@ install() { xfs_admin \ xfs_growfs \ wc \ - parted \ wipefs # growpart deps

overlay.d/05core/usr/libexec/coreos-ignition-firstboot-complete

Signed-off-by: Nikita Dubrovskii <[email protected]>

This brings Secure Execution support. Signed-off-by: Nikita Dubrovskii <[email protected]>

jlebon

Looks great, thanks!

jlebon · 2022-06-27T19:01:53Z

Hmm, interesting. It looks like the FCOS pipeline is failing to build s390x with this:

Importing rpm-md... done
rpm-md repo 'fedora-coreos-pool'; generated: 2022-06-26T16:28:46Z solvables: 17070
rpm-md repo 'fedora'; generated: 2022-05-04T21:15:53Z solvables: 57625
rpm-md repo 'fedora-updates'; generated: 2022-06-27T01:26:43Z solvables: 11906
error: Packages not found: /usr/bin/genprotimg

jlebon · 2022-06-27T19:07:31Z

Ahh yes of course, we missed a step here. We need to add s390utils-base to the s390x lockfile as documented in the README.

bgilbert previously requested changes Nov 29, 2021

View reviewed changes

nikita-dubrovskii mentioned this pull request Feb 4, 2022

s390x: add secure-execution support ostreedev/ostree#2491

Merged

nikita-dubrovskii force-pushed the secure-execution branch 2 times, most recently from 626f9eb to 0609f3e Compare February 16, 2022 16:37

nikita-dubrovskii force-pushed the secure-execution branch from 0609f3e to 49d018b Compare April 4, 2022 16:57

nikita-dubrovskii force-pushed the secure-execution branch from 49d018b to 320c0c4 Compare April 19, 2022 10:02

nikita-dubrovskii changed the title ~~s390x: add base secure-execution support~~ WIP: add base secure-execution support Apr 19, 2022

nikita-dubrovskii force-pushed the secure-execution branch 4 times, most recently from 00f916d to 194400a Compare April 22, 2022 10:45

nikita-dubrovskii force-pushed the secure-execution branch from 194400a to 0f7d60e Compare May 5, 2022 14:49

nikita-dubrovskii changed the title ~~WIP: add base secure-execution support~~ s390x: add secure-execution support May 5, 2022

nikita-dubrovskii force-pushed the secure-execution branch from 0f7d60e to fb401c5 Compare May 18, 2022 11:45

travier reviewed May 23, 2022

View reviewed changes

overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-growfs.sh Outdated Show resolved Hide resolved

overlay.d/05core/usr/lib/dracut/modules.d/35coreos-ignition/coreos-gpt-setup.sh Outdated Show resolved Hide resolved

nikita-dubrovskii force-pushed the secure-execution branch 2 times, most recently from e63b281 to 0b9b28a Compare May 24, 2022 14:15

jlebon reviewed May 24, 2022

View reviewed changes

jlebon reviewed May 25, 2022

View reviewed changes

overlay.d/05core/usr/lib/dracut/modules.d/35coreos-ignition/coreos-gpt-setup.sh Outdated Show resolved Hide resolved

nikita-dubrovskii force-pushed the secure-execution branch 2 times, most recently from 6bda640 to 5e4f879 Compare May 27, 2022 15:43

jlebon reviewed May 30, 2022

View reviewed changes

nikita-dubrovskii force-pushed the secure-execution branch 3 times, most recently from 6cd553a to 02808f0 Compare May 31, 2022 14:42

nikita-dubrovskii added 2 commits June 24, 2022 14:22

s390x: add 'genprotimg' tool to base image

858f84e

Signed-off-by: Nikita Dubrovskii <[email protected]>

s390x: coreos-gpt-setup: add suport for LUKS encrypted 'boot' partition

d44ee12

nikita-dubrovskii force-pushed the secure-execution branch from 02808f0 to 20789d2 Compare June 24, 2022 12:28

jlebon reviewed Jun 24, 2022

View reviewed changes

nikita-dubrovskii added 2 commits June 26, 2022 20:16

s390x: ignition-ostree-growfs: resize LUKS encrypted rootfs

0036fff

Signed-off-by: Nikita Dubrovskii <[email protected]>

coreos-ignition-firstboot-complete: call 'rdcore zipl' instead of 'zipl'

1d4931a

This brings Secure Execution support. Signed-off-by: Nikita Dubrovskii <[email protected]>

nikita-dubrovskii force-pushed the secure-execution branch from 20789d2 to 1d4931a Compare June 26, 2022 18:19

jlebon approved these changes Jun 27, 2022

View reviewed changes

jlebon merged commit ecc9497 into coreos:testing-devel Jun 27, 2022

nikita-dubrovskii deleted the secure-execution branch June 27, 2022 15:45

jlebon mentioned this pull request Jun 29, 2022

kola: s390x: ext.config.files.file-directory-permissions test fails coreos/fedora-coreos-tracker#1243

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

s390x: add secure-execution support #1353

s390x: add secure-execution support #1353

nikita-dubrovskii commented Nov 29, 2021 •

edited

Loading

bgilbert left a comment

travier left a comment

jlebon May 24, 2022

nikita-dubrovskii May 25, 2022

jlebon May 25, 2022

nikita-dubrovskii May 27, 2022

nikita-dubrovskii May 27, 2022

jlebon May 30, 2022

nikita-dubrovskii May 31, 2022

jlebon Jun 24, 2022

nikita-dubrovskii Jun 26, 2022

nikita-dubrovskii Jun 27, 2022

jlebon May 30, 2022

jlebon Jun 24, 2022

nikita-dubrovskii Jun 26, 2022

jlebon Jun 27, 2022

jlebon Jul 18, 2022

jlebon Jun 24, 2022

jlebon left a comment

jlebon commented Jun 27, 2022

jlebon commented Jun 27, 2022

s390x: add secure-execution support #1353

s390x: add secure-execution support #1353

Conversation

nikita-dubrovskii commented Nov 29, 2021 • edited Loading

bgilbert left a comment

Choose a reason for hiding this comment

travier left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlebon left a comment

Choose a reason for hiding this comment

jlebon commented Jun 27, 2022

jlebon commented Jun 27, 2022

nikita-dubrovskii commented Nov 29, 2021 •

edited

Loading