tls inspector makes mysql connection using tcp proxy timeout

[ferjjp]

Hi, this is my first issue on envoy!

I am using envoy as a sidecar transparent proxy and I have set up a rule that adds an envoy.filters.network.tcp_proxy that allows a container running a mysql client in the pod to access a mysql server. I was also using this listener for outgoing HTTP connections using HCM, and now a change in requirements requires me to have the tls inspector filter enabled on this listener.

After making the configuration changes we discovered that the outgoing mysql connections that we previously had were not working correctly:

[2022-04-25 19:09:44.960][26][debug][filter] [source/extensions/filters/listener/original_dst/original_dst.cc:20] original_dst: new connection accepted
[2022-04-25 19:09:44.960][26][debug][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:88] tls inspector: new connection accepted
[2022-04-25 19:09:45.013][1][debug][main] [source/server/server.cc:251] flushing stats
[2022-04-25 19:09:50.013][1][debug][main] [source/server/server.cc:251] flushing stats
[2022-04-25 19:09:55.013][1][debug][main] [source/server/server.cc:251] flushing stats
[2022-04-25 19:09:59.963][26][debug][conn_handler] [source/server/active_tcp_socket.cc:45] listener filter times out after 15000 ms
[2022-04-25 19:09:59.964][31][debug][filter] [source/extensions/filters/listener/original_dst/original_dst.cc:20] original_dst: new connection accepted
[2022-04-25 19:09:59.964][31][debug][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:88] tls inspector: new connection accepted

This is an extract of a mini-lab (using envoy 1.22) we used to isolate the problem. We discovered a timeout after 15000ms on the mysql connection. This timeout does not occur if the tls inspector listener filter is not used.

I looked around for any similar issues and found a few:

• An issue on aws app-mesh roadmap
• An issue that seems to have been fixed long ago but describes a problem that looks similar to mine

In this last issue, the problem was that mysql starts sending data to the client first. It seems to me that this is what's happening here, or something similar.

Simplified config that reproduces this:

admin:
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 12345
static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    listener_filters:
    - name: "envoy.filters.listener.tls_inspector"
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector
    filter_chains:
    - filter_chain_match:
      filters:
      - name: envoy.filters.network.tcp_proxy
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          cluster: mysql
          stat_prefix: mysql

  clusters:
  - name: mysql
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: mysql
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: mysql
                port_value: 3306

I'm creating this as a bug, but the thing is, I'm not sure if tls inspector should be working for this particular use case or not...Any input is appreciated, thanks.

[lambdai]

This case is the motivation of continue_on_listener_filters_timeout.

You can disable tls_inspector using filter_disabled. In your case, the filter_dsiabled should be configured to match mysql traffic but not match HCM traffic.

See https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/listener/v3/listener_components.proto#envoy-v3-api-msg-config-listener-v3-listenerfilter

[ferjjp]

I hadn't noticed the continue_on_listener_filters_timeout option at all. I'll try out the filter disabled option to see if I can make it match correctly, thank you!

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

[deveshkandpal1224]

@lambdai @alyssawilk - if you have a TCP tunneling usecase for mysql ( client -> envoy -> squid -> database ) where the tunneling_config depends on TLS inspector being present ( requested_server_name ) - is there a way to make this work still ?

[deveshkandpal1224]

it definitely works without tls_inspector but in that case I had to use %DOWNSTREAM_LOCAL_ADDRESS% in my tunneling_config to make this work. But would be nicer to get it to work with the reqested_server_name