Unable to connect to MySQL with App Mesh

[bcelenza]

Summary

When allowing egress traffic to all destinations (Mesh EgressFilter type=ALLOW_ALL), or specifically to a MySQL database via a VirtualNode definition, the connection will fail to MySQL with the error:

ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0

This error occurs because the egress listener in Envoy configured by App Mesh adds the TLS Inspector listener filter, which evaluates whether or not a connection is using TLS by inspecting the first packets sent from the client. However, in the case of MySQL, the server sends the first packets after connection. From that link:

Initial Handshake starts with server sending the Initial Handshake Packet.

Because the server sends the first packet, inspection at the filter fails.

The App Mesh team is currently investigating a fix for this issue.

Steps to reproduce

1. Create a Mesh with EgressFilter type ALLOW_ALL
2. Create a VirtualNode under the mesh. The VirtualNode does not require any special configuration.
3. Connect an Envoy to App Mesh's Envoy Management Service identified by the VirtualNode you created.
4. Attempt to connect to a network-reachable MySQL instance.

Expected: You are able to connect.
Actual: Connection fails with ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0

Workaround

To work around this issue, add port 3306 to the list of ignored egress ports for Envoy.

For ECS, you can set this in the Proxy Configuration.

For EKS, you can set the environment variable APPMESH_EGRESS_IGNORED_PORTS in the pod definition for the proxyinit container. See Getting Started with EKS for more information on this definition.

For EC2, you can set APPMESH_EGRESS_IGNORED_PORTS using the proxy route manager script. See Getting Started with EC2 for more information on this script.

[ryan-idea42]

For EKS, you can set the environment variable APPMESH_EGRESS_IGNORED_PORTS in the pod definition for the proxyinit container. See Getting Started with EKS for more information on this definition.

In my podspec, I added the following for proxyinit

      initContainers:
        - name: proxyinit
          image: 111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-proxy-route-manager:latest
          securityContext:
            capabilities:
              add: 
                - NET_ADMIN
          env:
            - name: "APPMESH_START_ENABLED"
              value: "1"
            - name: "APPMESH_IGNORE_UID"
              value: "1337"
            - name: "APPMESH_ENVOY_INGRESS_PORT"
              value: "15000"
            - name: "APPMESH_ENVOY_EGRESS_PORT"
              value: "15001"
            - name: "APPMESH_APP_PORTS"
              value: "80"
            - name: "APPMESH_EGRESS_IGNORED_PORTS"
              value: "3306,22"
            - name: "APPMESH_EGRESS_IGNORED_IP"
              value: "169.254.169.254"

When I deployed it, the pod fails to start and renders the following log:

=== Input configuration ===
APPMESH_EGRESS_IGNORED_PORTS=,22
APPMESH_IGNORE_UID=1337
APPMESH_START_ENABLED=1
APPMESH_APP_PORTS=80
APPMESH_ENVOY_INGRESS_PORT=15000
APPMESH_ENVOY_EGRESS_PORT=15001
APPMESH_EGRESS_IGNORED_IP=169.254.169.254
=== Initializing ===
iptables: Chain already exists.

It's almost like it's not using what I specify. Any idea on why it appears to ignore my ignored ports?

[bcelenza]

@ryan-idea42 You are correct. I've found a bug in our proxy manager script as well that prohibits this from being set appropriately. We're working on updating that container with the correct code, and will report back here once we have it published.

[bcelenza]

@ryan-idea42 Give 111345817488.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-proxy-route-manager:v2 a try -- should be fixed now. Sorry about that!

[ryan-idea42]

That did it! I'm able to hit the DB now and pull back results. Thank you!

[ryan-idea42]

@bcelenza Is this something that is permanent? Or is this just a temporary fix as we wait for an official fix?

Basically, wanting to know if we'll have issues with this in a production level environment.

[bcelenza]

@ryan-idea42 This is a temporary workaround while we work on an official fix. So long as you're okay not seeing MySQL traffic run though Envoy (for stats, etc.), this workaround can be used in production safely.

We'll respond back here once we have a plan in place for a longer term fix.

[rodrigopsasaki]

This helped me get out of a jam, spent a while chasing my tail on this one. I think the temporary fix should be documented somewhere as I'm sure I'm not the only one who has faced this situation as well

[dastbe]

To give ya'll an update, we're researching how to solve this from the perspective of Envoy for MySQL and other "server-sends-first-byte" protocols.

[LancerRainier]

We are now actively working on a change to enable MySQL traffic run though Envoy. We are expecting to complete the work here early April. A new AppMesh Envoy image will be available to enable this new feature. Stay tuned here.

[sergeyklay]

@LancerRainier

We are expecting to complete the work here early April.

But now is March. Do you mean April 2021?

[LancerRainier]

Thank for the response @sergeyklay. No, I did not mean April 2021. Isn't March earlier than April? Let me know if I'm missing anything here.

We are indeed actively working on this issue, and treating it as high priority as we know this is what AppMesh customer wants. Since we have a higher confidence on delivering this, thus making a rough time commitment here. Let me know if any questions.

Thank you!

[sergeyklay]

@LancerRainier Glad to see it get off the ground. I'm sorry it was bad joke

[LancerRainier]

@sergeyklay No worries! :)

[nealp9084]

Hi, it's April 2020 so we owe you a status update. We are actively working to fix this bug.

We are currently shoring up the AppMesh Envoy release process, which will ensure faster, higher-quality releases. This effort will pave the way for the AppMesh Envoy 1.14.x releases.

Once that's complete, we'll be able to start vending Envoy configurations that are compatible with MySQL databases (fixing the bug).

Update 5/5/2020: this should be fixed by early-mid Q3.

[icwr]

in case it helps somebody in the meantime, i was able to update the ignored ports by adding this annotation below in my deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: foo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      annotations:
        appmesh.k8s.aws/egressIgnoredPorts: "22,3306"
      labels:
        app: myapp

[LancerRainier]

A quick update since its been a while: We delayed this item due to various internal reasons, but we are actively making progress here to release it. We are in the release process and are looking at releasing this to all regions somewhere in early Q4 2020.

[shaileshgupta2k]

The issue has been fixed now for customers using App Mesh Envoy image version 1.15.0 or later. But, if you are still using an older version of AppMesh Envoy image, you still need to apply the quick-fix suggested at the top of this issue or in the official AWS document (link mentioned below) under the heading “Unable to connect to a MySQL or SMTP server”. If you need more details, please refer to the linked official AWS document. Thank you all for patiently waiting for the fix.

Official AWS Doc: https://docs.aws.amazon.com/app-mesh/latest/userguide/troubleshoot-connectivity.html