Conjur lookup plugin fails to verify TLS certificate - works ok in role and Conjur Go CLI

[jamesfreeman959]

Summary

I am trying to make use of the Conjur lookup plugin in a simple Ansible playbook. I have a Conjur OSS server 1.21.2-547 server and have been following some steps based on the tutorial for Ansible integration. So far, everything is working except the lookup plugin, which fails with: urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)>

Steps to Reproduce

1. Set up the Conjur CLI on the Ansible control node, init and log in (verify with conjur whoami)
2. Obtain a Host Factory token from the Conjur server
3. Run the cyberark.conjur.conjur_host_identity role to set up the Ansible control node
4. Run a simple playbook to query a known variable that has been stored in Conjur OSS
5. Playbook fails with an unhandled error - urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)>

Expected Results

I would expect this to work. The Conjur OSS server is behind a Traefik proxy, and has a valid LetsEncrypt (ACME) certificate issued. This certificate validates correctly in both the Conjur Go CLI tool, and the role, so it seems anomalous that it doesn't work in the lookup plugin.

Actual Results

I would expect the lookup plugin to return a valid value as both the Go CLI and the cyberark.conjur.conjur_host_identity are successfully validating the TLS certificate

Reproducible

• Always
• Sometimes
• Non-Reproducible

Version/Tag number

*CyberArk Conjur OSS 1.21.2-547

$ ansible --version
ansible [core 2.17.7]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/jamesf_local/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /home/jamesf_local/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.12.3 (main, Nov  6 2024, 18:32:19) [GCC 13.2.0] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True

Running on Ubuntu Server 24.04

Environment setup

• Conjur is running in a Docker container.
• Ansible is being run on a VM

Additional Information

Full output from Ansible:

Loading collection cyberark.conjur from /usr/lib/python3/dist-packages/ansible_collections/cyberark/conjur
conf file: /etc/conjur.conf
Loading configuration from: /etc/conjur.conf
identity file: /etc/conjur.identity
Loading identity from: /etc/conjur.identity for https://conjur.app.hkskies.com
Using cert file path /etc/conjur.pem
Authentication request to Conjur at: https://conjur.app.hkskies.com/authn/demo/host%2Fdemo.example.com/authenticate, with user: host%2Fdemo.example.com
exception during Jinja2 execution: Traceback (most recent call last):
  File "/usr/lib/python3.12/urllib/request.py", line 1344, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.12/http/client.py", line 1336, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.12/http/client.py", line 1382, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.12/http/client.py", line 1331, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.12/http/client.py", line 1091, in _send_output
    self.send(msg)
  File "/usr/lib/python3.12/http/client.py", line 1035, in send
    self.connect()
  File "/usr/lib/python3.12/http/client.py", line 1477, in connect
    self.sock = self._context.wrap_socket(self.sock,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/ssl.py", line 455, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/ssl.py", line 1042, in _create
    self.do_handshake()
  File "/usr/lib/python3.12/ssl.py", line 1320, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/ansible/template/__init__.py", line 827, in _lookup
    ran = instance.run(loop_terms, variables=self._available_variables, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/ansible_collections/cyberark/conjur/plugins/lookup/conjur_variable.py", line 419, in run
    token = _fetch_conjur_token(
            ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/ansible_collections/cyberark/conjur/plugins/lookup/conjur_variable.py", line 228, in _fetch_conjur_token
    response = open_url(conjur_url,
               ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/ansible/module_utils/urls.py", line 995, in open_url
    return Request().open(method, url, data=data, headers=headers, use_proxy=use_proxy,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/ansible/module_utils/urls.py", line 899, in open
    r = urllib.request.urlopen(request, None, timeout)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/urllib/request.py", line 515, in open
    response = self._open(req, data)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/urllib/request.py", line 532, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/lib/python3.12/urllib/request.py", line 1392, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/urllib/request.py", line 1347, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)>
fatal: [localhost]: FAILED! => {
    "msg": "An unhandled exception occurred while running the lookup plugin 'cyberark.conjur.conjur_variable'. Error was a <class 'urllib.error.URLError'>, original message: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)>. <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)>"
}

[jamesfreeman959]

Addendum - in case it's relevant, the Traefik proxy uses a wildcard certificate for all apps that it proxies, including Conjur.

I've confirmed that my code works fine when I add validate_certs=false so hopefully we can figure out a fix for this. I've had a look at the code but I can't see any reason why it's failing at the moment.

[jtuttle]

Thanks @jamesfreeman959, we will have someone take a look at this when time allows.

[jamesfreeman959]

Ok I have an answer (of sorts) - this might actually come down to an opportunity for documentation rather than an actual bug, and it's almost certainly also got something to do with how Python/Ansible handles CA certificate chains.

I followed the process discussed here to download the certificate chain from the Conjur server: https://discuss.cyberarkcommons.org/t/conjur-setup/1008

I've used similar before successfully, so I assumed (that was my error) it would work. On checking the contents of the PEM file generated by this command, I find that it's identical to the /etc/conjur.pem file which is generated by running the cyberark.conjur.conjur_host_identity module to grant a host access. So far so good, and this PEM file works with both this module, and the Conjur Go CLI.

What is interesting though is that this PEM file contains only two certificates. And if we refer to https://letsencrypt.org/certificates/ - we can see there should be 3 in the full chain. I can also see this in my browser if I navigate to the Conjur server and click on the padlock next to the URL.

Digging deeper into the PEM file generated, I noticed that the certificate for the ISRG Root X1 CA was missing. As a test, I downloaded this manually:

curl -O https://letsencrypt.org/certs/isrgrootx1.pem >> /etc/conjur.pem

Now the CA chain file contains all 3 certificates in the chain.

If I reset validate_certs to true in the playbook, it now works - all SSL errors have gone away. The Go CLI also still works. Haven't tested the module yet.

Also as a workaround, given that Ubuntu's included CA bundle can validate LetsEncrypt certificates, I tried:

export CONJUR_CERT_FILE=/etc/ssl/certs/ca-certificates.crt 

This also works perfectly, and should for any TLS certificate which has been signed by a well known CA.

Hope this information helps someone out!

[jtuttle]

Sounds like this will probably end up being a documentation improvement, as you say. Great stuff, thanks for digging in!