Consul Connect - Ability to configure envoy access_log_path

[prashantchari]

Feature Description

The access_log_path for Envoy currently defaults to /dev/null. It'd be great if this was configurable with Consul Connect.

Use Case(s)

Output envoy access logs to stdout or saving them to a local file.

[mkeeler]

Reopening to capture that we want to be able to configure the access log path for both the TCPProxy filters and the HTTP Connect Manager within the xDS package in addition to the access logs for the admin interface.

[stale]

Hey there,
We wanted to check in on this request since it has been inactive for at least 60 days.
If you think this is still an important issue in the latest version of Consul
or its documentation please reply with a comment here which will cause it to stay open for investigation.
If there is still no activity on this issue for 30 more days, we will go ahead and close it.

Feel free to check out the community forum as well!
Thank you!

[akhayyat]

Is there really no way today to enable and/or configure access logs for Envoy sidecars using Consul Connect?

[alkalinecoffee]

I don't think there is a way to simply pass in an access log endpoint other than /dev/null through native Consul config (although I think you can for admin logs, oddly enough).

However, there may be a way to pass it in using a Consul escape hatch which allows you to pass in custom Envoy-specific configuration directly to Envoy.

Using native Envoy configuration, this filter would send access logs to /dev/stdout:

static_resources:
  listeners:
  - name: inbound_10000
    address:
      socket_address:
        protocol: TCP
        address: 0.0.0.0
        port_value: 10000
    traffic_direction: INBOUND
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
          access_log:
          - name: envoy.access_loggers.file
            config:
              path: /dev/stdout

The escape hatch override config values map to a bootstrap template which is used to build the Envoy configuration file. You can find the template here: https://github.com/hashicorp/consul/blob/master/command/connect/envoy/bootstrap_tpl.go. You can also provide your own custom Envoy template using this same pattern and push standard log configurations to all of your proxies.

[pvyaka01]

Using escape hatch, I am able to get access logs. However, this becomes cumbersome when we're dealing with hundreds of proxies. The http_connection_manager filter requires address/socket/port_value and hard coding via escape hatch seems to be the only way at present. Access logs are very informative especially when we want to check specific upstream calls. Is there an easier why? Why not change the bootstrap to include this based on http or tcp protocols and not leave at default /dev/null? Thanks!

[akhayyat]

Would somebody please post working Consul configuration for enabling access logs of TCP envoy proxies?

[pvyaka01]

Something like this might help...

envoy_public_listener_json= <<EOL
             {
                "name": "local_agent",
                "address": {
                  "socket_address": {
                    "address": "0.0.0.0",
					"port_value": 20002
                  }
                },
                "filter_chains": [
                 {
                   "filters": [
                   {
                     "name": "envoy.tcp_proxy",
                     "config": {
					   "cluster": "redis01",
                       "stat_prefix": "envoy.tcp_proxy",
                     "tracing": {
                       "random_sampling": {}
                     },                     
                    "access_log": {
                      "name": "envoy.file_access_log",
                      "typed_config": {
                        "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
                        "path": "/var/log/envoy/envoy_tcp.log"
                     }
                    }
                  }
                }
              ]
            }
          ]
        }
        EOL

[pvyaka01]

Did that help?

[TsvetanMilanov]

Any plans to support access log path for HttpConnectionManager with configuration like the admin access log path?

[evandam]

Hey folks, any chance this would be coming up in a future release? It would be super helpful to be able to configure access logging centrally for L7 services and ingress gateways where it's more important to see things like client IP, without having to resort to escape hatches.

[dizzler]

Bumping this request and adding some context from a security perspective.

It is important to make it easy for customer's to retrieve "Access Logs" from Envoy.
Envoy produces two types of data for proxied connections: 1) stats & 2) access logs.
Different use-cases apply to the different types of data.
The stats are great for aggregating the massive volume of proxy data, while access logs provide an incredible level of detail for individual connections.

From a security perspective, nothing compares to access logs (i.e. aggregated metrics/stats are not enough).
One of the main challenges in network security monitoring is that most traffic is encrypted end-to-end, especially in cloud environments. You cannot (usefully) inspect what you cannot decrypt. Service meshes (like Consul Connect) change this dynamic, where most (if not all) traffic is inspected (authenticated, authorized, logged) before being encrypted.

Most customers will not want to enable Envoy access logs anywhere, while a minority of customers will want to enable Envoy access logs everywhere, then feed these access logs to their log monitoring (SIEM, SOAR, etc.) solution. For those that want "universal observability" at network Layer-7, Consul Connect should support a "global" config option for enabling Envoy access logs on all HTTP and TCP connection managers. This config option should accept a string value that sets the output path for Envoy access logs, where the value sets the path of a local output file and the default is "stdout".

For comparison -- and apologies in advance for referring to another project -- this feature is already supported in Istio:
https://istio.io/latest/docs/tasks/observability/logs/access-log/#enable-envoy-s-access-logging

[pvyaka01]

I completely agree but seems there's no traction on this, unfortunately. We rely on access logs quite a bit.

[davidesalerno]

I'm facing the same issue (configure the ingress gateway to write the HTTP access log so that we can forward it to Splunk) and I would like to understand if my thoughts are feasible.

Using the envoy_public_listener_json (https://www.consul.io/docs/connect/proxies/envoy#envoy_public_listener_json) is there a way to pass a public listener with a filter in the chain that will configure the listener to log the HTTP requests in the stdout?

In addition, our Consul setup is on Kubernetes so should we do it using a global ProxyDefaults CRD?

I agree with the fact that this feature should be available via Consul like many other products are doing (Istio, AWS App Mesh, Ambassador, Gloo etc) because it is really important especially in an "Enterprise" context.

[narendrapatel]

Hi, Is there any updates to this. At our side, access logging is an important part of debugging.
Right now we've configured logging for inbound listener on VMs to a file with help to escape hatch overrides. But can't use the same for other listeners since escape hatches conflict with usage of configs like service-router/splitter/resolver.
Also difficult to use escape hatches in Kubernetes. Have made a FR here : #12040

Ability to configure logging with access log formatting would be really helpful.

[porterctrlz]

Any movement here? Little shocking that this isn't configurable.

[david-yu]

Hi all we anticipate closing by our next release 1.15, to allow you to globally enable and disable access logs via the proxyDefaults Config entry/CRD. Hope this helps.

[hyungwook0221]

Hi @david-yu.
Can we also collect access log through Ingress gateway in version 1.15?
Also, I wonder if the access log from Consul API Gateway can be applied.

[david-yu]

Hi @hyungwook0221 we intend to support access logs for proxies, mesh gateways, ingress gateways, terminating gateways and Consul API Gateway.

[itaytalmi]

What would be the proper way to enable access logging in Consul 1.14 (Envoy 1.23)? I have tried the above suggestions (using native Envoy configuration, an escape hatch, listeners, bootstrap_tpl, etc.) but can't get it to work. Is there any temporary workaround? Even service-level configuration (not global) would significantly help at this point.

[david-yu]

Access logs is now available in Consul 1.15.0, and will also be supported in K8s for Consul K8s 1.1.0. #15864. We will not be backporting the change to 1.14.x and below as it is not backportable.

Will close this issue and open a new issue for Access Logs support for API Gateway since that was not addressed by that PR.

[david-yu]

Here is the issue for access logs support for API Gateway: #16419