repository_set cannot find the repository

[runephilosof-karnovgroup]

SUMMARY

repository_set cannot find the repository
Hammer can find it and see that it is enabled

[xxx@xxx ~]# hammer repository-set info --id 7416 --organization-id 1
Id:                   7416
Name:                 Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)
Type:                 yum
Url:                  /content/dist/rhel8/$releasever/x86_64/baseos/os
GPG Key:              file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Label:                rhel-8-for-x86_64-baseos-rpms
Enabled Repositories: 
 1) Id:   3
    Name: Red Hat Enterprise Linux 8 for x86_64 - BaseOS RPMs 8

ISSUE TYPE

• Bug Report

ANSIBLE VERSION

ansible [core 2.16.6]
  config file = /home/xxx/code/platform-satellite/foreman-config/ansible.cfg
  configured module search path = ['/home/xxx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.12/site-packages/ansible
  ansible collection location = /home/xxx/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.12.3 (main, Apr 17 2024, 00:00:00) [GCC 13.2.1 20240316 (Red Hat 13.2.1-7)] (/usr/bin/python3)
  jinja version = 3.1.3
  libyaml = True

COLLECTION VERSION

# /usr/lib/python3.12/site-packages/ansible_collections
Collection                               Version
---------------------------------------- -------
amazon.aws                               7.5.0  
ansible.netcommon                        5.3.0  
ansible.posix                            1.5.4  
ansible.utils                            2.12.0 
ansible.windows                          2.3.0  
arista.eos                               6.2.2  
awx.awx                                  23.9.0 
azure.azcollection                       1.19.0 
check_point.mgmt                         5.2.3  
chocolatey.chocolatey                    1.5.1  
cisco.aci                                2.9.0  
cisco.asa                                4.0.3  
cisco.dnac                               6.13.3 
cisco.intersight                         2.0.8  
cisco.ios                                5.3.0  
cisco.iosxr                              6.1.1  
cisco.ise                                2.8.1  
cisco.meraki                             2.18.0 
cisco.mso                                2.6.0  
cisco.nxos                               5.3.0  
cisco.ucs                                1.10.0 
cloud.common                             2.1.4  
cloudscale_ch.cloud                      2.3.1  
community.aws                            7.2.0  
community.azure                          2.0.0  
community.ciscosmb                       1.0.7  
community.crypto                         2.19.0 
community.digitalocean                   1.26.0 
community.dns                            2.9.0  
community.docker                         3.9.0  
community.general                        8.6.0  
community.grafana                        1.8.0  
community.hashi_vault                    6.2.0  
community.hrobot                         1.9.2  
community.library_inventory_filtering_v1 1.0.1  
community.libvirt                        1.3.0  
community.mongodb                        1.7.3  
community.mysql                          3.9.0  
community.network                        5.0.2  
community.okd                            2.3.0  
community.postgresql                     3.4.0  
community.proxysql                       1.5.1  
community.rabbitmq                       1.3.0  
community.routeros                       2.15.0 
community.sap                            2.0.0  
community.sap_libs                       1.4.2  
community.sops                           1.6.7  
community.vmware                         4.3.0  
community.windows                        2.2.0  
community.zabbix                         2.3.1  
containers.podman                        1.13.0 
cyberark.conjur                          1.2.2  
cyberark.pas                             1.0.25 
dellemc.enterprise_sonic                 2.4.0  
dellemc.openmanage                       8.7.0  
dellemc.powerflex                        2.3.0  
dellemc.unity                            1.7.1  
f5networks.f5_modules                    1.28.0 
fortinet.fortimanager                    2.4.0  
fortinet.fortios                         2.3.6  
frr.frr                                  2.0.2  
gluster.gluster                          1.0.2  
google.cloud                             1.3.0  
grafana.grafana                          2.2.5  
hetzner.hcloud                           2.5.0  
hpe.nimble                               1.1.4  
ibm.qradar                               2.1.0  
ibm.spectrum_virtualize                  2.0.0  
ibm.storage_virtualize                   2.3.1  
infinidat.infinibox                      1.4.5  
infoblox.nios_modules                    1.6.1  
inspur.ispim                             2.2.0  
inspur.sm                                2.3.0  
junipernetworks.junos                    5.3.1  
kubernetes.core                          2.4.2  
lowlydba.sqlserver                       2.3.2  
microsoft.ad                             1.5.0  
netapp.aws                               21.7.1 
netapp.azure                             21.10.1
netapp.cloudmanager                      21.22.1
netapp.elementsw                         21.7.0 
netapp.ontap                             22.11.0
netapp.storagegrid                       21.12.0
netapp.um_info                           21.8.1 
netapp_eseries.santricity                1.4.0  
netbox.netbox                            3.17.0 
ngine_io.cloudstack                      2.3.0  
ngine_io.exoscale                        1.1.0  
openstack.cloud                          2.2.0  
openvswitch.openvswitch                  2.1.1  
ovirt.ovirt                              3.2.0  
purestorage.flasharray                   1.27.0 
purestorage.flashblade                   1.17.0 
purestorage.fusion                       1.6.1  
sensu.sensu_go                           1.14.0 
splunk.es                                2.1.2  
t_systems_mms.icinga_director            2.0.1  
telekom_mms.icinga_director              1.35.0 
theforeman.foreman                       3.15.0 
vmware.vmware_rest                       2.3.1  
vultr.cloud                              1.12.1 
vyos.vyos                                4.1.0  
wti.remote                               1.0.5  

# /usr/share/ansible/collections/ansible_collections
Collection                               Version
---------------------------------------- -------
ansible.netcommon                        4.1.0  
ansible.posix                            1.5.4  
ansible.utils                            2.9.0  
chocolatey.chocolatey                    1.5.1  
community.docker                         3.4.6  
community.general                        8.0.2  
community.kubernetes                     2.0.1  
community.libvirt                        1.3.0  
community.mysql                          3.7.2  
community.rabbitmq                       1.2.3  
containers.podman                        1.12.0 
fedora.linux_system_roles                1.78.1 
google.cloud                             1.0.2  
kubernetes.core                          2.3.2  
microsoft.sql                            2.0.2  
netbox.netbox                            3.9.0  

KATELLO/FOREMAN VERSION

package tfm-rubygem-katello is not installed
foreman-3.10.0-1.el9.noarch
katello-4.12.0-1.el9.noarch

STEPS TO REPRODUCE

- name: "Enable repository RHEL 8 BaseOS"
  theforeman.foreman.repository_set:
    username: "admin"
    password: "{{ foreman_admin_password }}"
    server_url: "https://{{ ansible_host }}"
    organization: "{{ organization }}"
    label: rhel-8-for-x86_64-baseos-rpms
    repositories:
      - releasever: "8"

EXPECTED RESULTS

The repository_set is already enabled, so it should just be ok

ACTUAL RESULTS

TASK [xxx_foreman_2 : Enable repository RHEL 8 BaseOS] **************************************************************************************************************************************************************
fatal: [xxx]: FAILED! => {"changed": false, "msg": "Desired repositories are not available on the repository set label: rhel-8-for-x86_64-baseos-rpms.\nSearched: {'label': 'rhel-8-for-x86_64-baseos-rpms', 'repositories': [{'releasever': '8'}]}\nFound: []\nAvailable: []"}

[evgeni]

Can you show us the output of curl -u admin:password https://foreman.example.com/katello/api/repository_sets/7416/available_repositories?

[runephilosof-karnovgroup]

curl -u admin:password https://xxx/katello/api/repository_sets/7416/available_repositories?organization_id=1
# {"total":0,"subtotal":0,"selectable":0,"page":null,"per_page":null,"error":null,"search":null,"sort":{"by":null,"order":null},"results":[]}

[evgeni]

"Good" (for some values of good, at least).

This says that your Katello thinks there are no repositories available. That can have multiple reasons: connection problems reaching cdn.redhat.com, expired Manifest, expired Subscriptions (among others). Could you have a look at /var/log/foreman/production.log around the time you perform the curl, are there any errors or warnings?

[runephilosof-karnovgroup]

requesting https://xxx/katello/api/repository_sets/7416/available_repositories?organization_id=1 gives the following error in the production.log

2024-05-22T09:58:53 [I|app|432a8b3c] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel8/listing
2024-05-22T09:58:53 [E|app|402c9519] Failed at scanning for repository: CDN loading error: access forbidden to https://cdn.redhat.com:443/content/dist/rhel8/listing

I must have made some error, when I converted the manifest to be ansible controlled, removing access to them.

However, it is confusing that the web ui shows them as available on https://xxx/redhat_repositories with the selector set to available and recommended.

https://xxx/katello/api/v2/repository_sets?organization_id=1&with_active_subscription=true&search=(((name+!~+source+rpm)+and+(name+!~+debug+rpm)+and+(content_type+%3D+yum)+and+(label+!~+beta)+and+(label+!~+htb)+and+(name+!~+beta)+and+(product_name+!~+beta)))+and+(label+%3D+rhel-9-for-x86_64-baseos-rpms+or+label+%3D+rhel-9-for-x86_64-baseos-kickstart+or+label+%3D+rhel-9-for-x86_64-appstream-rpms+or+label+%3D+rhel-9-for-x86_64-appstream-kickstart+or+label+%3D+rhel-8-for-x86_64-baseos-rpms+or+label+%3D+rhel-8-for-x86_64-baseos-kickstart+or+label+%3D+rhel-8-for-x86_64-appstream-rpms+or+label+%3D+rhel-8-for-x86_64-appstream-kickstart+or+label+%3D+rhel-8-for-x86_64-baseos-eus-rpms+or+label+%3D+rhel-8-for-x86_64-appstream-eus-rpms+or+label+%3D+rhel-7-server-rpms+or+label+%3D+rhel-7-server-optional-rpms+or+label+%3D+rhel-7-server-extras-rpms+or+label+%3D+rhel-7-server-kickstart+or+label+%3D+satellite-client-6-for-rhel-9-x86_64-rpms+or+label+%3D+satellite-client-6-for-rhel-8-x86_64-rpms+or+label+%3D+rhel-7-server-satellite-client-6-rpms+or+label+%3D+rhel-6-server-els-satellite-client-6-rpms+or+label+%3D+satellite-capsule-6.15-for-rhel-8-x86_64-rpms+or+label+%3D+satellite-maintenance-6.15-for-rhel-8-x86_64-rpms+or+label+%3D+satellite-utils-6.15-for-rhel-8-x86_64-rpms)

returns 12 results, among others the rhel-8-for-x86_64-baseos-rpms.

When making the above query the production.log does not contain CDN: Requesting path

[evgeni]

I think the UI just assumes enabled repos to be available, and thus "tricks" you in the view.
I am honestly not sure what's more correct here.
Given the repository is actually enabled, you could argue the "enable it" action would be a noop and thus we should consider it "available", on the other hand there is clearly an issue with your setup and you might not have noticed otherwise?
(I would expect syncs of that repo to fail too, btw)

[runephilosof-karnovgroup]

Yes, the syncs were also failing.

I clicked the refresh button in the manage manifest modal which made it all work again.
Very confusing, especially because the ui says nothing about what refreshing the manifest means and when it will be necessary to do.

It does not seem like anything needs to be changed in the foreman-ansible-modules, unless you want to suggest refreshing the manifest in the error message.

[evgeni]

I don't think changing the error message makes sense (there are many different reasons why repos can't be found, this is just one of them)

But I pinged @jeremylenz whether Katello could more prominently expose the need for a refresh in the UI.