With v5.1.0 AAP/AWX inventory syncs fail

[2and3makes23]

SUMMARY

Since v5.1.0 Ansible Automation Platform/AWX inventory syncs using the foreman inventory plugin do no longer work.
When pinning theforeman.foreman to v5.0.0 there is no error.

ISSUE TYPE

• Bug Report

ANSIBLE VERSION

ansible-core 2.17.5

COLLECTION VERSION

theforeman.foreman v5.1.0

KATELLO/FOREMAN VERSION

Katello 4.11.1/Foreman 3.9.3

STEPS TO REPRODUCE

In AWX/AAP setup an inventory using the Red Hat Satellite 6 as Source

As filter we use

---
host_filters: 'name ~ thehostname and hostgroup ~ Testing'

EXPECTED RESULTS

Inventory Sync succeeds, as in:

ansible-inventory [core 2.17.5]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.12/site-packages/ansible
  ansible collection location = /runner/.ansible/collections:/usr/share/ansible/collections:/usr/share/automation-controller/collections
  executable location = /usr/local/bin/ansible-inventory
  python version = 3.12.6 (main, Sep  9 2024, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-2)] (/usr/bin/python3.12)
  jinja version = 3.1.4
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
[DEPRECATION WARNING]: ANSIBLE_COLLECTIONS_PATHS option, does not fit var 
naming standard, use the singular form ANSIBLE_COLLECTIONS_PATH instead. This 
feature will be removed from ansible-core in version 2.19. Deprecation warnings
 can be disabled by setting deprecation_warnings=False in ansible.cfg.
Using inventory plugin 'ansible_collections.theforeman.foreman.plugins.inventory.foreman' to process inventory source '/runner/inventory/foreman.yml'
[WARNING]: Failed to use Reports API, falling back to Hosts API: Error
scheduling inventory report on foreman. Please check foreman logs!
Parsed /runner/inventory/foreman.yml inventory source with auto plugin
    5.643 INFO     Processing JSON output...
    5.647 INFO     Loaded 38 groups, 18 hosts
    6.535 INFO     Inventory import completed for thehost_k in 0.9s

ACTUAL RESULTS

Sync fails:

AWX Output:

ansible-inventory [core 2.17.5]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.12/site-packages/ansible
  ansible collection location = /runner/.ansible/collections:/usr/share/ansible/collections:/usr/share/automation-controller/collections
  executable location = /usr/local/bin/ansible-inventory
  python version = 3.12.6 (main, Sep  9 2024, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-2)] (/usr/bin/python3.12)
  jinja version = 3.1.4
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
[DEPRECATION WARNING]: ANSIBLE_COLLECTIONS_PATHS option, does not fit var 
naming standard, use the singular form ANSIBLE_COLLECTIONS_PATH instead. This 
feature will be removed from ansible-core in version 2.19. Deprecation warnings
 can be disabled by setting deprecation_warnings=False in ansible.cfg.
Using inventory plugin 'ansible_collections.theforeman.foreman.plugins.inventory.foreman' to process inventory source '/runner/inventory/foreman.yml'
[WARNING]:  * Failed to parse /runner/inventory/foreman.yml with auto plugin:
Error scheduling inventory report on foreman. Please check foreman logs!
  File "/usr/local/lib/python3.12/site-packages/ansible/inventory/manager.py", line 292, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/usr/local/lib/python3.12/site-packages/ansible/plugins/inventory/auto.py", line 58, in parse
    plugin.parse(inventory, loader, path, cache=cache)
  File "/usr/share/ansible/collections/ansible_collections/theforeman/foreman/plugins/inventory/foreman.py", line 690, in parse
    self._populate()
  File "/usr/share/ansible/collections/ansible_collections/theforeman/foreman/plugins/inventory/foreman.py", line 447, in _populate
    self._populate_report_api()
  File "/usr/share/ansible/collections/ansible_collections/theforeman/foreman/plugins/inventory/foreman.py", line 479, in _populate_report_api
    host_data = copy.deepcopy(self._post_request())
                              ^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/ansible/collections/ansible_collections/theforeman/foreman/plugins/inventory/foreman.py", line 427, in _post_request
    raise Exception("Error scheduling inventory report on foreman. Please check foreman logs!")
[WARNING]: Unable to parse /runner/inventory/foreman.yml as an inventory source
ERROR! No inventory was parsed, please check your configuration and options.

Foreman production.log

2024-12-13T11:18:13 [I|app|3bf4c337] Started GET "/api/v2/status?page=1&per_page=250" for <ip> at 2024-12-13 11:18:13 +0100
2024-12-13T11:18:13 [I|app|3bf4c337] Processing by Api::V2::HomeController#status as JSON
2024-12-13T11:18:13 [I|app|3bf4c337]   Parameters: {"page"=>"1", "per_page"=>"250", "apiv"=>"v2"}
2024-12-13T11:18:13 [I|app|3bf4c337] Authorized user api-dyn-inventory(dynamic inventory user)
2024-12-13T11:18:13 [I|app|3bf4c337]   Rendered api/v2/home/status.json.rabl (Duration: 0.3ms | Allocations: 159)
2024-12-13T11:18:13 [I|app|3bf4c337] Completed 200 OK in 152ms (Views: 0.8ms | ActiveRecord: 16.4ms | Allocations: 14037)
2024-12-13T11:18:13 [I|app|6ddc1adc] Started POST "/ansible/api/v2/ansible_inventories/schedule" for <ip> at 2024-12-13 11:18:13 +0100
2024-12-13T11:18:13 [F|app|6ddc1adc]
 6ddc1adc | ActionController::RoutingError (No route matches [POST] "/ansible/api/v2/ansible_inventories/schedule"):
 6ddc1adc |
 6ddc1adc | lib/foreman/middleware/logging_context_request.rb:11:in `call'
 6ddc1adc | katello (4.11.1) lib/katello/prevent_json_parsing.rb:12:in `call' 

ADDITIONAL DETAILS

Most probably this behavior is tied to a bug fix in 5.1.0: inventory - Drop fallback to Host API when Reports API fails, as this leads to possibly wrong data being used

Could you please provide information on how to best solve this issue?
As it seems, the way it worked prior to 5.1.0 is not optimal either (bugfix). So we´d rather find a way to make it work with versions >= v5.1.0

Thanks so much! :)

[evgeni]

Correct, that's due to removal of that fallback.

All you need to do is to explicitly disable the Reports API in your config:

foreman-ansible-modules/tests/inventory/inventory_plugin.foreman.yml

Line 8 in 5f5ba20

use_reports_api: false