feat: bulild vsphere template in air gapped environment

[supershal]

What problem does this PR solve?:

• create vsphere image template for RHEL 7.9 and 8.4 in air gapped environments

Which issue(s) does this PR fix?:

• https://jira.d2iq.com/browse/D2IQ-84576

Output:
Following template was created in airgapped VM network.
The packer job to create image in template was run from a bastion VM on the same airgapped network as the VM.

--> vsphere-clone: konvoy-ova-vsphere-rhel-79-1.21.6-1645637142
--> vsphere-clone: konvoy-ova-vsphere-rhel-79-1.21.6-1645637142

Special notes for your reviewer:

issues and workarounds:

• Issue: The cloud-init datasource for vmware guestinfo was merged in Aug 2021 (Datasource for VMware canonical/cloud-init#953) and only available in cloud-init > 21.3 . The cloud-init rpms that are available in Centos7/RHEL repos is > 19.4-7.

  • workaround: upstream project downloads and runs the installation script from the archived repo (https://raw.githubusercontent.com/vmware/cloud-init-vmware-guestinfo/v1.4.0/install.sh) to install the vmware datasource. similar workaround is done to support this in airgapped mode where we copy the datasource module from ansible host to the guest and then run the script.

• issue: The vmware cloudinit datasource module requires 8.1>pip<21.6 for Centos7/RHEL7. The pip-8.1.2 is default version. The latest pip version breaks on the Centos7/RHEL7

  • workaround: Download fix latest pip version 20.3.4 that is < 21.6 for offline builds

• issue: unable to run packer job for airgapped environment from laptop or teamcity job as the VMs created in airgapped environments are not accessible from outside.

  • workaround: Created a bastion node manually that has two VM networks: "public" and "airgapped". copy the repository to the bastion host using public IP. then run packer to create image in "airgapped" environment by overriding networking for the target VM to "airgapped"
  • enhancements: Create a terraform script to create bastion host with two network interfaces. copy konvoy-image binary and ansible/packer files to the bastion host and run packer remotely with ssh. (similar to e2e tests in konvoy2)
    should we embed ansible playbooks in the konvoy-binary too?

[supershal]

Makefile target clutter is getting worse. I will have a PR in next sprint to fix this.

[supershal]

copied from upstream to make it work in offline mode. PR description for details. any suggestion for better way to make this work?

[github-actions]

File	Coverage
All files	3%	❌
pkg/ansible/runner.go	0%	❌
pkg/app/artifacts.go	0%	❌
pkg/app/build.go	7%	❌
pkg/app/errors.go	0%	❌
pkg/app/provision.go	0%	❌
pkg/app/root.go	0%	❌
pkg/app/validate.go	0%	❌
pkg/appansible/io.go	0%	❌
pkg/appansible/playbook.go	0%	❌
pkg/logging/logger.go	0%	❌
pkg/packer/manifest.go	0%	❌
pkg/packer/packer.go	0%	❌
pkg/stringutil/rand.go	0%	❌
pkg/version/info.go	0%	❌

Minimum allowed coverage is 75%

Generated by 🐒 cobertura-action against f8fdfb7

[dkoshkin]

Will users still be able to have their own cloud-init?

[supershal]

Theoretically yes. However it would require an extra setup outside our normal flow : Create template using KIB -> Extra steps: (convert template to VM, change VM config go add userdata using UI or govc CLI or API), convert vm back to template) -> reference template name in DKP CLI when creating cluster.
govc command: https://github.com/mevansam/vsphere-govmomi/blob/master/govc/USAGE.md#vmchange

   govc vm.change -vm "${VM}" \
  -e guestinfo.metadata="${METADATA}" \
  -e guestinfo.metadata.encoding="gzip+base64" \
  -e guestinfo.userdata="${USERDATA}" \
  -e guestinfo.userdata.encoding="gzip+base64"

The cluster-api uses it to configure VM using data from KubeadmConfigTemplate
https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/blob/master/pkg/services/govmomi/extra/config.go#L42

[jkoelker]

Its not needed now, since even RHEL9 only has 21.1 and the provider is in 21.3 but should we check if ${PY_MOD_CLOUD_INIT}/sources/DataSourceVMware.py exists, and then bail early?

[supershal]

This is a good change, as this will prevent us from overriding latest DS whenever it is available. I have made this change

[supershal]

@jkoelker I had to change dependency order of kubeadm role and packages role since kubeadm installs latest version of kubectl rpm which is not compatible with kubeadm. This needs to fixed upstream, but as a stop gap I had to provide fix ordering between these two roles. Please let me know if this hack is fine.

[jkoelker]

@jkoelker I had to change dependency order of kubeadm role and packages role since kubeadm installs latest version of kubectl rpm which is not compatible with kubeadm. This needs to fixed upstream, but as a stop gap I had to provide fix ordering between these two roles. Please let me know if this hack is fine.

:sad-panda:, yep that should be fine, the list-images.yaml playbook only needs kubeadm and containerd roles, so changing the order on the provision.yaml playbook should work no problems.