Could not execute posix command: No such file or directory -

[maxadamo]

Description of problem

• when puppet runs in background it throws an error in the syslog, when it's being run from the CLI this error doesn't show.

This is the error that I have observed the logs:

Could not execute posix command: No such file or directory -

• The error appears on a number of machines (not all) and I'm not able to state the difference between these VMs.
  I am not using agent_entity_config_provider, in any case (this is important, see below) hence, it should default to API.

• I enabled trace and debug to and I found out that the problem was caused by a provider in the Sensu module, attempting to run sensuctl.
  The sensu provider is this one:

#  ll /opt/puppetlabs/puppet/cache/lib/puppet/provider/sensuctl.rb
-rw-r--r-- 1 root root 6204 Jan 28 00:02 /opt/puppetlabs/puppet/cache/lib/puppet/provider/sensuctl.rb

• Trace output:

Feb 05 17:55:22 repositories02 puppet-agent[4673]: Could not execute posix command: No such file or directory -
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/execution.rb:360:in `exec'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/execution.rb:360:in `block (2 levels) in execute_posix'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:128:in `withenv'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/execution.rb:359:in `block in execute_posix'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:527:in `block in safe_posix_fork'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:508:in `fork'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:508:in `safe_posix_fork'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/execution.rb:323:in `execute_posix'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/execution.rb:217:in `execute'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider.rb:106:in `execute'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/cache/lib/puppet/provider/sensuctl.rb:74:in `sensuctl'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/cache/lib/puppet/provider/sensuctl.rb:94:in `sensuctl_list'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/cache/lib/puppet/provider/sensu_namespace/sensuctl.rb:13:in `instances'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/cache/lib/puppet/provider/sensu_namespace/sensuctl.rb:25:in `prefetch'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:378:in `prefetch'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:260:in `prefetch_if_necessary'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:115:in `block in evaluate'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:120:in `traverse'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:178:in `evaluate'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `block (2 levels) in apply'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:237:in `block in apply'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:236:in `apply'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:193:in `block (2 levels) in apply_catalog'
Feb 05 17:55:22 repositories02 puppet-agent[4673]: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'

[maxadamo]

I installed sensuctl and the problem vanished, but I should not to do this, and only on some VM: the twin VM, is repositories01 and it doesn't have this issue.

[treydock]

Based on that stack trace you have sensu_namespace defined for the host so Puppet is evaluating which namespaces exist and it does this by default using sensuctl. The way Puppet works in general, and this module relies on this, the instances function of a provider is only executed on a host where that host has some resource defined that uses the provider. I would recommend looking in /opt/puppetlabs/puppet/cache/state/resources.txt on the hosts with this problem and check to see if those hosts have a sensu_namespace resource defined. In most cases only a sensu-backend host should have sensu resources like sensu_namespace defined in their catalog.

[maxadamo]

I believe, that since I have:
validate_namespaces => false, then, sensu_namespace becomes mandatory, otherwise I get another error (that I can show).

[maxadamo]

If I don't use sensu_namespace I get this error (because of validate_namespace => false):

Error: /Stage[main]/Sensu::Agent/Sensu::Agent::Subscription[lg]/Sensu_agent_entity_config[sensu::agent::subscription lg]: Sensu namespace 'default' must be defined or exist
Error: /Stage[main]/Sensu::Agent/Sensu::Agent::Subscription[devops_linux]/Sensu_agent_entity_config[sensu::agent::subscription devops_linux]: Sensu namespace 'default' must be defined or exist
Error: /Stage[main]/Sensu::Agent/Sensu::Agent::Subscription[linux_metrics]/Sensu_agent_entity_config[sensu::agent::subscription linux_metrics]: Sensu namespace 'default' must be defined or exist
Error: /Stage[main]/Sensu::Agent/Sensu::Agent::Subscription[vmware]/Sensu_agent_entity_config[sensu::agent::subscription vmware]: Sensu namespace 'default' must be defined or exist
Error: /Stage[main]/Sensu::Agent/Sensu::Agent::Label[host_name]/Sensu_agent_entity_config[sensu::agent::label host_name]: Sensu namespace 'default' must be defined or exist
Error: /Stage[main]/Sensu::Agent/Sensu::Agent::Label[fqdn]/Sensu_agent_entity_config[sensu::agent::label fqdn]: Sensu namespace 'default' must be defined or exist
Error: /Stage[main]/Sensu::Agent/Sensu::Agent::Label[ip_address]/Sensu_agent_entity_config[sensu::agent::label ip_address]: Sensu namespace 'default' must be defined or exist
Error: /Stage[main]/Sensu::Agent/Sensu::Agent::Label[ip6_address]/Sensu_agent_entity_config[sensu::agent::label ip6_address]: Sensu namespace 'default' must be defined or exist
Error: /Stage[main]/Sensu::Agent/Sensu::Agent::Label[domain_password]/Sensu_agent_entity_config[sensu::agent::label domain_password]: Sensu namespace 'default' must be defined or exist
Error: /Stage[main]/Sensu::Agent/Sensu::Agent::Label[nagios_check_path]/Sensu_agent_entity_config[sensu::agent::label nagios_check_path]: Sensu namespace 'default' must be defined or exist
Error: /Stage[main]/Sensu::Agent/Sensu::Agent::Label[metrics_label]/Sensu_agent_entity_config[sensu::agent::label metrics_label]: Sensu namespace 'default' must be defined or exist
Error: Failed to apply catalog: Some pre-run checks failed

[maxadamo]

to be clear: sensu_namespace is enabled everywhere, but only in some case I see the error:
Could not execute posix command: No such file or directory -

[treydock]

That error you get because of validate_namespace being false is actually because the agent API is unable to verify the namespace you specified exists. If you run puppet locally on one of those hosts and pass the --debug flag like puppet agent -t --debug then you would see messages like ERROR fetching namespaces via Sensu API and that might have some clues as to why it's failing.

There could be many reasons for that, the most common is you don't have sensu::agent_entity_config_password set for both backends and agents to the same value. That is the password the agent Puppet code will use to talk to the backend API. The other reason could be you don't have sensu::api_host set to point to your backend server, that is also used by the agent Puppet code to connect to the backend API.

If both of those are set for both agents and backends, then the next issue could be the agent simply can not communicate with the backend API. You can test that by on the agent doing something like this:

curl -k https://backend.example.com:8080/version

Adjust as needed for your backend environment.

[treydock]

Also you should not have sensu_namespace defined on agents, because then every single agent thinks it's responsible for a central resource stored on the backend. If you go to change the namespace in some way on one agent, other agents might change it back. It's best to only allow a single host to be responsible for a specific resource, and in most cases you only want to define sensu_* resources on the backend host. The only ones that must be defined on agent are sensu_agent_entity_config.

[maxadamo]

Sensu API is reachable everywhere (it's a virtualhost, on haproxy, with Keepalived: it's always up)

#  curl https://sensuapi.XXXX.org:443/version
{"etcd":{"etcdserver":"3.3.22","etcdcluster":"3.3.0"},"sensu_backend":"6.2.3"}

What I didn't like, to be honest, is that Sensu became a single point of failure for puppet: if the agent is not being able to contact the API host, puppet hangs, and fails. I have a cluster with 3 nodes, but I could not set the API to use the 3 nodes, and I didn't want to have the agents, failing for this reason.
Anyway, I'll give immediately a try

[maxadamo]

it seems to be working if I use validate_namespaces => true and I unset sensu_namespace,
But then, there is a parameter which cannot be used.

[treydock]

We have it on our roadmap to support an array for api_host so that the API code will try each host until one works. In your case if you have your backends behind HAProxy, why not point api_host at the HAProxy address so that the current single value api_host is less likely to fail if one backend is down?

What parameter cannot be used? Please clarify.

[maxadamo]

The parameter that I'm not able to use it
validate_namespaces => false, with sensu_namespace, or am I misunderstanding the use case of this parameter?

[treydock]

The Puppet code by default will check to see if a resource's defined namespace exists on the backend. So if you define a sensu_check with namespace 'test', then by default the code will check that either 1) the namespace 'test' exists on the backend or 2) the namespace 'test' is defined using sensu_namespace. The #2 check only works when the same host defines the resource like sensu_check and the sensu_namespace, which is one of the many reasons it's best to define resources on the backend host.

The purpose of validate_namespace was to bypass this check in situations where someone defines thousands of resources that all end up validating their namespace. The validation done thousands of times on a host could dramatically slow down Puppet applying the catalog, so validate_namespace is a way to avoid that slow down.

[maxadamo]

but, in that case (I use your words, the case where someone defines thousands of resource that all end up validating their namespace) it won't work, if sensuctl is not available.
Are you sure this is not a problem for you ❓

[treydock]

Those thousands of resources would all be defined on the backend, and the way this module works is the backend hosts always have sensuctl. The slow down with namespace validation is when a single catalog contains thousands of resources, not when thousand of agents each validate a namespace for a resource. The way this module is designed is most sites should be defining all resources on the backend host, and agents only define sensu_agent_entity_config resources.

[maxadamo]

Understood. Thanks.