delegate_to leads to SSL verification error

[sscheib]

SUMMARY

Using delegate_to in a task with any theforeman.foreman.* module leads to SSL connection issues:

"msg": "Failed to connect to Foreman server: SSLError: HTTPSConnectionPool(host='host.example.com', port=443): Max retries exceeded with url: /api/status (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))"
}

Verification of the SSL connection using openssl s_client -connect -showcerts host.example.com </dev/null validates perfectly:

[..]
---
SSL handshake has read 8465 bytes and written 438 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
[..]

ISSUE TYPE

• Bug Report

ANSIBLE VERSION

ansible [core 2.14.5]
  config file = /home/user/.ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.9/site-packages/ansible
  ansible collection location = /home/user/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.9.13 (main, Nov  9 2022, 13:16:24) [GCC 8.5.0 20210514 (Red Hat 8.5.0-15)] (/usr/bin/python3.9)
  jinja version = 3.1.2
  libyaml = True

COLLECTION VERSION

# /home/user/.ansible/collections/ansible_collections
Collection                  Version
--------------------------- -------
theforeman.foreman          3.10.0

KATELLO/FOREMAN VERSION

# rpm -q tfm-rubygem-katello foreman
package tfm-rubygem-katello is not installed
foreman-3.3.0.21-2.el8sat.noarch

STEPS TO REPRODUCE

---
- name: 'Configure OpenSCAP'
  hosts: 'all'
  gather_facts: false
  vars:
    scap_content_download_path: '/tmp'
    scap_content_files:
      - name: 'DISA STIG RHEL 8 (Version 1, Release 9)'
        url: 'https://dl.dod.cyber.mil/wp-content/uploads/stigs/zip/U_RHEL_8_V1R9_STIG_SCAP_1-2_Benchmark.zip'

      - name: 'DISA STIG RHEL 7 (Version 1, Release 11)'
        url: 'https://dl.dod.cyber.mil/wp-content/uploads/stigs/zip/U_RHEL_7_V3R11_STIG_SCAP_1-2_Benchmark.zip'

  tasks:
    - name: 'Download and unpack OpenSCAP content'
      ansible.builtin.unarchive:
        src: '{{ item.url }}'
        dest: '{{ scap_content_download_path }}'
        mode: 0600
        remote_src: true
      loop: '{{ scap_content_files }}'
      delegate_to: 'localhost'

    - name: 'Create OpenSCAP content'
      theforeman.foreman.scap_content:
        username: '{{ satellite_username }}'
        password: '{{ satellite_password }}'
        server_url: '{{ satellite_server_url }}'
        validate_certs: true
        scap_file: "{{ scap_content_download_path ~ '/' ~ item.url | basename | regex_replace('.zip$', '.xml') }}"
        state: 'present'
        title: '{{ item.name }}'
      delegate_to: 'localhost'
      loop: '{{ scap_content_files }}

As theforeman.foreman.scap_content needs scap_file, I need to delegate to localhost as the files only will be temporarily downloaded to /tmp and afterwards removed of the control node.
Note: I am not running in an execution environment, still using ansible-playbook

EXPECTED RESULTS

SCAP contents are created

ACTUAL RESULTS

SSL verification fails. This is only observed when using delegate_to. As seen above, the SSL connection validated successfully, though.

$ ansible-playbook -i inventory openscap.yml --vault-pass-file .vault.pass
PLAY [Configure OpenSCAP] ***************************************************************************************************************************************************************

TASK [Download and unpack OpenSCAP content] *********************************************************************************************************************************************
ok: [host.example.com -> localhost] => (item={'name': 'DISA STIG RHEL 8 (Version 1, Release 9)', 'url': 'https://dl.dod.cyber.mil/wp-content/uploads/stigs/zip/U_RHEL_8_V1R9_STIG_SCAP_1-2_Benchmark.zip'})
ok: [host.example.com -> localhost] => (item={'name': 'DISA STIG RHEL 7 (Version 1, Release 11)', 'url': 'https://dl.dod.cyber.mil/wp-content/uploads/stigs/zip/U_RHEL_7_V3R11_STIG_SCAP_1-2_Benchmark.zip'})

TASK [Create OpenSCAP content] **********************************************************************************************************************************************************
failed: [host.example.com -> localhost] (item={'name': 'DISA STIG RHEL 8 (Version 1, Release 9)', 'url': 'https://dl.dod.cyber.mil/wp-content/uploads/stigs/zip/U_RHEL_8_V1R9_STIG_SCAP_1-2_Benchmark.zip'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "DISA STIG RHEL 8 (Version 1, Release 9)", "url": "https://dl.dod.cyber.mil/wp-content/uploads/stigs/zip/U_RHEL_8_V1R9_STIG_SCAP_1-2_Benchmark.zip"}, "msg": "Failed to connect to Foreman server: SSLError: HTTPSConnectionPool(host='host.example.com', port=443): Max retries exceeded with url: /api/status (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))"}
failed: [host.example.com -> localhost] (item={'name': 'DISA STIG RHEL 7 (Version 1, Release 11)', 'url': 'https://dl.dod.cyber.mil/wp-content/uploads/stigs/zip/U_RHEL_7_V3R11_STIG_SCAP_1-2_Benchmark.zip'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "DISA STIG RHEL 7 (Version 1, Release 11)", "url": "https://dl.dod.cyber.mil/wp-content/uploads/stigs/zip/U_RHEL_7_V3R11_STIG_SCAP_1-2_Benchmark.zip"}, "msg": "Failed to connect to Foreman server: SSLError: HTTPSConnectionPool(host='host.example.com', port=443): Max retries exceeded with url: /api/status (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))"}

PLAY RECAP ******************************************************************************************************************************************************************************
host.example.com : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

[evgeni]

I'm using hosts: localhost vs delegate_to: localhost here, and that works. AFAIK those should be identical?

As it's working for me, couple of questions

1. What is localhost in this context?
2. Which Python is used when you execute against localhost?
3. If that's not the system Python, it might use the certs bundled in certifi which then ignores your system-wide installed certs that openssl uses?

[sscheib]

Thanks for looking into this!
I use 'all' for hosts as I provide variables via host_vars for the specific Satellite instances (configuring multiple)

What is localhost in this context?

The local controller node, which is a plain RHEL 8.7. Not running in an execution environment, so it is really referring to that machine.

Which Python is used when you execute against localhost?

It's the same as with hosts: 'all':

"ansible_python_interpreter": "/usr/bin/python3.9"

If that's not the system Python, it might use the certs bundled in certifi which then ignores your system-wide installed certs that openssl uses?

It's the same in both cases

[sscheib]

I cannot reproduce this issue anymore with the latest release. Either it was an issue on my end or it was unknowingly fixed with later version
Anyway, closing it as complete.

[sscheib]

I know, this is an old issue.

For anybody stumbling on this issue, I finally managed to reproduce it.
The issue is present in a Python virtual environment when custom certificates are used.

This can be fixed with: https://stackoverflow.com/questions/34931378/certificate-verification-when-using-virtual-environments

[evgeni]

Thanks for tracking this down!