Add /agent/pprof endpoint

[schmichael]

Summary

Add /agent/pprof endpoint which is the equivalent of Go's /debug/pprof endpoint but is:

1. Enabled by default when ACLs are enabled and enable_debug is not explicitly set to false.
2. Protected by agent:write ACL
3. Supports ?node=... routing to client nodes via RPC (same as nomad monitor CLI #6365)

ACL

While agent:read is the most obvious choice for this API, due to its power and low-level access it should require a higher level of access than other agent:read endpoints like agent/members.

This also helps protect against future changes to the underlying Go package exposes more information than we currently expect.

As it's very difficult to determine all of the ways this Go runtime data could be used by an attacker, using a high access level is appropriate.

Default behavior

The default behavior is unfortunately confusing but seeks to maximize security, backward compatibility, and still allow debug access by default when possible. Disabling the debug endpoint by default makes it unavailable when operators need it most: when encountering an unexpected and hard to diagnose issue.

Endpoint	enable_debug	ACLs	Available?
/debug/pprof	unset	n/a	no
/debug/pprof	true	n/a	yes
/debug/pprof	false	n/a	no
/agent/pprof	unset	off	no
/agent/pprof	unset	on	yes
/agent/pprof	true	off	yes
/agent/pprof	false	n/a	no

Deprecating old endpoint behavior

The old endpoint is insecure and its use should be discouraged. However, it is useful in Nomad development scenarios where -dev is used or ACLs are disabled. A new configuration parameter should be added to control /debug/pprof:

insecure_enable_debug = true # defaults to false

In 0.11 setting enable_debug = true should log an INFO line pointing to this future behavior change:

In 0.12 setting enable_debug = true will no longer enable the unauthenticated /debug/pprof endpoint.

Consul & Vault

In Vault 1.3 the endpoint will be enabled by default but behind ACLs (which are always enabled in Vault).

Consul does not enable the endpoint by default but always requires an ACL.

[drewbailey]

@schmichael for the table of scenarios /s /debug/pprof with /v1/agent/pprof/ and /insecure_enable_debug controls the old agent only /debug/pprof endpoint?

[drewbailey]

/agent/pprof endpoints added in 0.10.4 https://github.com/hashicorp/nomad/blob/master/CHANGELOG.md#0104-february-19-2020

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.