Collection's util resource discovery fails when complex subresources present #659

dkasanic · 2023-11-15T13:44:38Z

SUMMARY

Collection's util resource discovery fails on Python Value error due to subresources having multiple "/" in string.

Kubevirt introduces such subresources, take a look here: https://github.com/kubevirt/kubevirt/blob/main/pkg/virt-api/api.go#L638
However, discovery util in collections expects all subresources having only 1 "/" char in string, look here: https://github.com/ansible-collections/kubernetes.core/blob/main/plugins/module_utils/client/discovery.py#L116

ISSUE TYPE

Bug Report

COMPONENT NAME

plugins/module_utils/client/discover.py

ANSIBLE VERSION

ansible [core 2.15.6]
  config file = /home/dkasanix/deployment/sdle/ansible.cfg
  configured module search path = ['/home/dkasanix/deployment/sdle/library']
  ansible python module location = /home/dkasanix/deployment/sdle/venv/lib/python3.10/site-packages/ansible
  ansible collection location = /home/dkasanix/deployment/sdle/collections
  executable location = /home/dkasanix/deployment/sdle/venv/bin/ansible
  python version = 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (/home/dkasanix/deployment/sdle/venv/bin/python)
  jinja version = 3.1.2
  libyaml = True

COLLECTION VERSION

# /home/dkasanix/deployment/sdle/venv/lib/python3.10/site-packages/ansible_collections
Collection      Version
--------------- -------
kubernetes.core 2.4.0

CONFIGURATION

CACHE_PLUGIN(/home/dkasanix/deployment/sdle/ansible.cfg) = jsonfile
CACHE_PLUGIN_CONNECTION(/home/dkasanix/deployment/sdle/ansible.cfg) = /tmp
CACHE_PLUGIN_TIMEOUT(/home/dkasanix/deployment/sdle/ansible.cfg) = 7200
CALLBACKS_ENABLED(/home/dkasanix/deployment/sdle/ansible.cfg) = ['timer', 'profile_tasks', 'profile_roles']
COLLECTIONS_PATHS(/home/dkasanix/deployment/sdle/ansible.cfg) = ['/home/dkasanix/deployment/sdle/collections']
CONFIG_FILE() = /home/dkasanix/deployment/sdle/ansible.cfg
DEFAULT_ACTION_PLUGIN_PATH(/home/dkasanix/deployment/sdle/ansible.cfg) = ['/home/dkasanix/deployment/sdle/action_plugins', '/home/dkasanix/.ansible/plugins/action', '/usr/share/ansible/plugins/action']
DEFAULT_GATHERING(/home/dkasanix/deployment/sdle/ansible.cfg) = smart
DEFAULT_LOG_PATH(/home/dkasanix/deployment/sdle/ansible.cfg) = /home/dkasanix/deployment/sdle/.ansible_last_run.log
DEFAULT_MODULE_PATH(/home/dkasanix/deployment/sdle/ansible.cfg) = ['/home/dkasanix/deployment/sdle/library']
DEFAULT_ROLES_PATH(/home/dkasanix/deployment/sdle/ansible.cfg) = ['/home/dkasanix/deployment/sdle/roles', '/home/dkasanix/deployment/sdle/collections/share/roles']
DEFAULT_STDOUT_CALLBACK(/home/dkasanix/deployment/sdle/ansible.cfg) = debug
DISPLAY_ARGS_TO_STDOUT(/home/dkasanix/deployment/sdle/ansible.cfg) = False
DISPLAY_SKIPPED_HOSTS(/home/dkasanix/deployment/sdle/ansible.cfg) = False
HOST_KEY_CHECKING(/home/dkasanix/deployment/sdle/ansible.cfg) = False
TRANSFORM_INVALID_GROUP_CHARS(/home/dkasanix/deployment/sdle/ansible.cfg) = ignore

OS / ENVIRONMENT

Kubernetes 1.28.3 with Calico CNI.
Kubevirt installed.

STEPS TO REPRODUCE

install kubevirt via https://kubevirt.io/user-guide/operations/installation/#installing-kubevirt-on-kubernetes
run task using module from kubernetes.core collection that happens to use discovery utils:

- name: test k8s_info
  kubernetes.core.k8s_info:
    kind: Deployment
    namespace: kubevirt

EXPECTED RESULTS

Task should return list of deployment objects present in kubevirt namespace.

ACTUAL RESULTS

Task fails on python Value error:

TASK [test] *****************************************************************************************************************************************************************************************************task path: /home/dkasanix/deployment/sdle/test.yml:9
The full traceback is:
Traceback (most recent call last):
  File "<stdin>", line 107, in <module>
  File "<stdin>", line 99, in _ansiballz_main
  File "<stdin>", line 47, in invoke_module
  File "/usr/lib/python3.10/runpy.py", line 224, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,                                                                                                                                                                     File "/usr/lib/python3.10/runpy.py", line 86, in _run_code                                                                                                                                                         exec(code, run_globals)                                                                                                                                                                                        File "/tmp/ansible_kubernetes.core.k8s_info_payload_4mk5e9ds/ansible_kubernetes.core.k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py", line 217, in <module>                File "/tmp/ansible_kubernetes.core.k8s_info_payload_4mk5e9ds/ansible_kubernetes.core.k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py", line 211, in main                    File "/tmp/ansible_kubernetes.core.k8s_info_payload_4mk5e9ds/ansible_kubernetes.core.k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py", line 173, in execute_module          File "/tmp/ansible_kubernetes.core.k8s_info_payload_4mk5e9ds/ansible_kubernetes.core.k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/service.py", line 228, in find            File "/tmp/ansible_kubernetes.core.k8s_info_payload_4mk5e9ds/ansible_kubernetes.core.k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/service.py", line 90, in find_resource    File "/tmp/ansible_kubernetes.core.k8s_info_payload_4mk5e9ds/ansible_kubernetes.core.k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/client.py", line 306, in resource         File "/tmp/ansible_kubernetes.core.k8s_info_payload_4mk5e9ds/ansible_kubernetes.core.k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/client.py", line 286, in _find_resource_with_prefix                                                                                                                                                                                                        File "/tmp/ansible_kubernetes.core.k8s_info_payload_4mk5e9ds/ansible_kubernetes.core.k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/client/discovery.py", line 159, in get
  File "/usr/local/lib/python3.10/dist-packages/kubernetes/dynamic/discovery.py", line 237, in search                                                                                                                results = self.__search(self.__build_search(**kwargs), self.__resources, [])                                                                                                                                   File "/usr/local/lib/python3.10/dist-packages/kubernetes/dynamic/discovery.py", line 283, in __search                                                                                                              matches.extend(self.__search([key] + parts[1:], resources, reqParams))                                                                                                                                         File "/usr/local/lib/python3.10/dist-packages/kubernetes/dynamic/discovery.py", line 269, in __search                                                                                                              return self.__search(parts[1:], resourcePart, reqParams + [part] )
  File "/usr/local/lib/python3.10/dist-packages/kubernetes/dynamic/discovery.py", line 283, in __search                                                                                                              matches.extend(self.__search([key] + parts[1:], resources, reqParams))
  File "/usr/local/lib/python3.10/dist-packages/kubernetes/dynamic/discovery.py", line 269, in __search
    return self.__search(parts[1:], resourcePart, reqParams + [part] )
  File "/usr/local/lib/python3.10/dist-packages/kubernetes/dynamic/discovery.py", line 258, in __search
    resourcePart.resources = self.get_resources_for_api_version(
  File "/tmp/ansible_kubernetes.core.k8s_info_payload_4mk5e9ds/ansible_kubernetes.core.k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/client/discovery.py", line 117, in get_resources_for_api_version
ValueError: too many values to unpack (expected 2)
fatal: [ad07-13-cyp]: FAILED! => {
    "changed": false,
    "invocation": {
        "kind": "Deployment",
        "module_args": {
            "kind": "Deployment",
            "namespace": "kubevirt"
        },
        "namespace": "kubevirt"
    },
    "rc": 1
}

MSG:

MODULE FAILURE
See stdout/stderr for the exact error


MODULE_STDERR:

Traceback (most recent call last):
  File "<stdin>", line 107, in <module>
  File "<stdin>", line 99, in _ansiballz_main
  File "<stdin>", line 47, in invoke_module
  File "/usr/lib/python3.10/runpy.py", line 224, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_kubernetes.core.k8s_info_payload_4mk5e9ds/ansible_kubernetes.core.k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py", line 217, in <module>
  File "/tmp/ansible_kubernetes.core.k8s_info_payload_4mk5e9ds/ansible_kubernetes.core.k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py", line 211, in main
  File "/tmp/ansible_kubernetes.core.k8s_info_payload_4mk5e9ds/ansible_kubernetes.core.k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py", line 173, in execute_module
  File "/tmp/ansible_kubernetes.core.k8s_info_payload_4mk5e9ds/ansible_kubernetes.core.k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/service.py", line 228, in find
  File "/tmp/ansible_kubernetes.core.k8s_info_payload_4mk5e9ds/ansible_kubernetes.core.k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/service.py", line 90, in find_resource
  File "/tmp/ansible_kubernetes.core.k8s_info_payload_4mk5e9ds/ansible_kubernetes.core.k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/client.py", line 306, in resource
  File "/tmp/ansible_kubernetes.core.k8s_info_payload_4mk5e9ds/ansible_kubernetes.core.k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/client.py", line 286, in _find_resource_with_prefix
  File "/tmp/ansible_kubernetes.core.k8s_info_payload_4mk5e9ds/ansible_kubernetes.core.k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/client/discovery.py", line 159, in get
  File "/usr/local/lib/python3.10/dist-packages/kubernetes/dynamic/discovery.py", line 237, in search
    results = self.__search(self.__build_search(**kwargs), self.__resources, [])
  File "/usr/local/lib/python3.10/dist-packages/kubernetes/dynamic/discovery.py", line 283, in __search
    matches.extend(self.__search([key] + parts[1:], resources, reqParams))
  File "/usr/local/lib/python3.10/dist-packages/kubernetes/dynamic/discovery.py", line 269, in __search
    return self.__search(parts[1:], resourcePart, reqParams + [part] )
  File "/usr/local/lib/python3.10/dist-packages/kubernetes/dynamic/discovery.py", line 283, in __search
    matches.extend(self.__search([key] + parts[1:], resources, reqParams))
  File "/usr/local/lib/python3.10/dist-packages/kubernetes/dynamic/discovery.py", line 269, in __search
    return self.__search(parts[1:], resourcePart, reqParams + [part] )
  File "/usr/local/lib/python3.10/dist-packages/kubernetes/dynamic/discovery.py", line 258, in __search
    resourcePart.resources = self.get_resources_for_api_version(
  File "/tmp/ansible_kubernetes.core.k8s_info_payload_4mk5e9ds/ansible_kubernetes.core.k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/client/discovery.py", line 117, in get_resources_for_api_version
ValueError: too many values to unpack (expected 2)

The text was updated successfully, but these errors were encountered:

bjwyun · 2023-11-16T07:14:27Z

Hi team, met the same issue in discovery.py
the api group subresources.kubevirt.io has some subresources

{'kind': '',
 'name': 'virtualmachineinstances/sev/querylaunchmeasurement',
 'namespaced': True,
 'singularName': '',
 'verbs': None}, {'kind': '',
 'name': 'virtualmachineinstances/sev/setupsession',
 'namespaced': True,
 'singularName': '',
 'verbs': None}, {'kind': '',
 'name': 'virtualmachineinstances/sev/injectlaunchsecret',
 'namespaced': True,
 'singularName': '',
 'verbs': None}

the name have two '/' in it.
but https://github.com/ansible-collections/kubernetes.core/blob/main/plugins/module_utils/client/discovery.py#L116

resource, name = subresource["name"].split("/")

only give two params to get value. it run into the same issue like

a,b = [1,2,3]
ValueError: too many values to unpack (expected 2)

@dkasanic the workaround is adding api_version= "apps/v1" it will find the deployment directly without interate all the resource groups.so you will not touch subresources.kubevirt.io

- name: test k8s_info
  kubernetes.core.k8s_info:
    api_version: "apps/v1"
    kind: Deployment
    namespace: kubevirt

bastienbosser · 2024-01-31T16:02:05Z

Hi,
We have the same problem and on our side we have fixed the code, specifying to the split to do it only on the first /.

resource, name = subresource["name"].split("/", 1)

It seems more logical to add this control, since the current code doesn't support more than two elements

…ces present ansible-collections#659)

liranr23 · 2024-02-28T15:06:27Z

We are dealing with the same error but on other occasion. In our case we try to post a configmap which is only v1 as the apiVersion.
We are executing:

- name: "Setup controller config map"
    k8s:
      state : present
      definition: "{{ lookup('template', 'controller/configmap-controller.yml.j2') }}"

And the full configmap is:

---
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: {{ app_name }}
    service: {{ controller_service_name }}
    control-plane: controller-manager
    controller-tools.k8s.io: "1.0"
  name: {{ controller_configmap_name }}
  namespace: {{ app_namespace }}
data:
  WORKING_DIR: {{ inventory_volume_path }}
{% if controller_precopy_interval is number %}
  PRECOPY_INTERVAL: "{{ controller_precopy_interval }}"
{% endif %}
{% if controller_max_vm_inflight is number %}
  MAX_VM_INFLIGHT: "{{ controller_max_vm_inflight }}"
{% endif %}

And the error:

 TASK [Setup controller config map] ********************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ValueError: too many values to unpack (expected 2)
fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"/opt/ansible/.ansible/tmp/ansible-tmp-1709132503.686507-1101-140623711667405/AnsiballZ_k8s.py\", line 102, in <module>\n    _ansiballz_main()\n  File \"/opt/ansible/.ansible/tmp/ansible-tmp-1709132503.686507-1101-140623711667405/AnsiballZ_k8s.py\", line 94, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/opt/ansible/.ansible/tmp/ansible-tmp-1709132503.686507-1101-140623711667405/AnsiballZ_k8s.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.community.kubernetes.plugins.modules.k8s', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib64/pyt...
-------------------------------------------------------------------------------
{"level":"error","ts":"2024-02-28T15:01:44Z","logger":"logging_event_handler","msg":"","name":"forklift-controller","namespace":"openshift-mtv","gvk":"forklift.konveyor.io/v1beta1, Kind=ForkliftController","event_type":"runner_on_failed","job":"1903436280976775946","EventData.Task":"Setup controller config map","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/forkliftcontroller/tasks/main.yml:55","error":"[playbook task failed]","stacktrace":"github.com/operator-framework/operator-sdk/internal/ansible/events.loggingEventHandler.Handle\n\toperator-sdk/internal/ansible/events/log_events.go:111"}
{"level":"error","ts":"2024-02-28T15:01:44Z","logger":"runner","msg":"\u001b[0;34mansible-playbook 2.9.27\u001b[0m\r\n\u001b[0;34m  config file = /etc/ansible/ansible.cfg\u001b[0m\r\n\u001b[0;34m  configured module search path = ['/usr/share/ansible/openshift']\u001b[0m\r\n\u001b[0;34m  ansible python module location = /usr/lib/python3.6/site-packages/ansible\u001b[0m\r\n\u001b[0;34m  executable location = /usr/bin/ansible-playbook\u001b[0m\r\n\u001b[0;34m  python version = 3.6.8 (default, Jan  5 2024, 08:58:17) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)]\u001b[0m\r\n\u001b[0;34mUsing /etc/ansible/ansible.cfg as config file\u001b[0m\r\n\u001b[0;34mSkipping callback 'actionable', as we already have a stdout callback.\u001b[0m\n\u001b[0;34mSkipping callback 'awx_display', as we already have a stdout callback.\u001b[0m\n\u001b[0;34mSkipping callback 'counter_enabled', as we already have a stdout callback.\u001b[0m\n\u001b[0;34mSkipping callback 'debug', as we already have a stdout callback.\u001b[0m\n\u001b[0;34m...
----- Ansible Task Status Event StdOut (forklift.konveyor.io/v1beta1, Kind=ForkliftController, forklift-controller/openshift-mtv) -----

We started seeing it only on a newer version of k8s. It started on 1.28. My guess is something changed in the returned value from k8s.

…ces present ansible-collections#659)

…ces present #659) (#676) * fix(Collection's util resource discovery fails when complex subresources present #659) * fix(add changelog fragment) * update node image * Create discovery.yml * Update main.yml --------- Co-authored-by: Bastien Bosser <[email protected]> Co-authored-by: Bikouo Aubin <[email protected]>

…ces present #659) (#676) * fix(Collection's util resource discovery fails when complex subresources present #659) * fix(add changelog fragment) * update node image * Create discovery.yml * Update main.yml --------- Co-authored-by: Bastien Bosser <[email protected]> Co-authored-by: Bikouo Aubin <[email protected]> (cherry picked from commit 1955989)

…ces present #659)

…discovery fails when complex subresources present #659) (#687) * fix(Collection's util resource discovery fails when complex subresources present #659) (#676) * fix(Collection's util resource discovery fails when complex subresources present #659) * fix(add changelog fragment) * update node image * Create discovery.yml * Update main.yml --------- Co-authored-by: Bastien Bosser <[email protected]> Co-authored-by: Bikouo Aubin <[email protected]> (cherry picked from commit 1955989) * adding sanity ignore file * Remove refresh_ignore_files * Update ignore files content --------- Co-authored-by: bastienbosser <[email protected]> Co-authored-by: abikouo <[email protected]>

…e discovery fails when complex subresources present #659) (#688) * update node image * fix(Collection's util resource discovery fails when complex subresources present #659) * fix(add changelog fragment) * Create discovery.yml * Update main.yml * Backport PR659 * fix sanity issues --------- Co-authored-by: Bastien Bosser <[email protected]>

gravesm added type/bug Something isn't working jira labels Nov 16, 2023

e3b0c442 mentioned this issue Dec 17, 2023

ValueError: too many values to unpack (expected 2) on [installer : Check for presence of old awx Deployment] ansible/awx-operator#1671

Closed

3 tasks

David-Igou mentioned this issue Jan 29, 2024

Redis fails to deploy due to "ValueError: too many values to unpack (expected 2)" ansible/eda-server-operator#174

Closed

bastienbosser pushed a commit to bastienbosser/kubernetes.core that referenced this issue Jan 31, 2024

fix(Collection's util resource discovery fails when complex subresour…

273663d

…ces present ansible-collections#659)

bastienbosser mentioned this issue Jan 31, 2024

fix(Collection's util resource discovery fails when complex subresources present #659) #676

Merged

abikouo pushed a commit to abikouo/kubernetes.core that referenced this issue Feb 28, 2024

fix(Collection's util resource discovery fails when complex subresour…

5df4194

…ces present ansible-collections#659)

gravesm pushed a commit to bastienbosser/kubernetes.core that referenced this issue Feb 29, 2024

fix(Collection's util resource discovery fails when complex subresour…

796e636

…ces present ansible-collections#659)

gravesm mentioned this issue Feb 29, 2024

Dynamic discovery cannot handle APIs of format a/b/c #685

Closed

abikouo closed this as completed Feb 29, 2024

patchback bot mentioned this issue Feb 29, 2024

[PR #676/19559892 backport][stable-3] fix(Collection's util resource discovery fails when complex subresources present #659) #687

Merged

abikouo pushed a commit that referenced this issue Mar 1, 2024

fix(Collection's util resource discovery fails when complex subresour…

5d6e63b

…ces present #659)

David-Igou mentioned this issue Jun 2, 2024

Cannot create postgresql instance due to "\nValueError: too many values to unpack (expected 2)\n" ansible/galaxy-operator#128

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Collection's util resource discovery fails when complex subresources present #659

Collection's util resource discovery fails when complex subresources present #659

dkasanic commented Nov 15, 2023

bjwyun commented Nov 16, 2023

bastienbosser commented Jan 31, 2024

liranr23 commented Feb 28, 2024

Collection's util resource discovery fails when complex subresources present #659

Collection's util resource discovery fails when complex subresources present #659

Comments

dkasanic commented Nov 15, 2023

SUMMARY

ISSUE TYPE

COMPONENT NAME

ANSIBLE VERSION

COLLECTION VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

bjwyun commented Nov 16, 2023

bastienbosser commented Jan 31, 2024

liranr23 commented Feb 28, 2024