App Mesh Envoy Authorization Policies

[bcelenza]

App Mesh will soon be releasing support for enabling TLS on Virtual Nodes in a Mesh (#39). As part of this change, we’re adding an additional authorization capability in App Mesh’s Envoy Management Service that will allow you to scope down the permissions for the IAM policy associated with your Envoy Proxy.

Starting today, you can add a new statement to your IAM policy giving Envoy Proxy explicit access to retrieve configuration from Envoy Management Service.

/* Example IAM policy statement for Envoy Management Service authorization */
{
    "Effect": "Allow",    
    "Action": ["appmesh:StreamAggregatedResources"],
    /* Optionally a set of VirtualNode ARNs */
    "Resource": "*" 
}

Am I required to make this change?

You will be initially required to add this policy statement to use TLS on a Virtual Node with a certificate provided by AWS Certificate Manager.

If the appmesh:StreamAggregatedResources action is not present, or the Virtual Node name used to identify the Envoy Proxy does not match the contents of the Resource property of the policy statement, then Envoy Proxy will be disconnected with gRPC code 7 (PERMISSION_DENIED) when attempting to retrieve the certificate materials from Envoy Management Service.

On November 5th, 2019, App Mesh will require this change for all configuration provided by Envoy Management Service. The reason for this is to provide a more secure and consistent experience for all customers using IAM for authorization to specific resources.

We recommend you add this policy statement as soon as possible, even if you do not plan on using TLS for your Virtual Nodes.

How can I opt-in to this change?

If you would like to opt-in to this change, you can adjust your IAM policy for the IAM User or Role you use to connect Envoy Proxy to App Mesh’s Envoy Management Service.

For example, if you’re using ECS with App Mesh, your current task role for the ECS task running the Envoy Proxy might look like:

# Example ECS Task Role definition in CloudFormation
TaskIamRole:
    Type: AWS::IAM::Role
    Properties:
      Path: /
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service:
                - "ecs-tasks.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/CloudWatchFullAccess
        - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess

To update this policy to support fetching certificate materials from App Mesh, you have two options detailed below.

Option 1: Add an Explicit Policy Statement (Recommended)

You can add an explicit statement to your IAM policy for the appmesh:StreamAggregatedResources action. We recommend this option because you can restrict which Virtual Nodes your Envoy may retrieve configuration for.

The following example shows a policy scoped down to only allow retrieval of configuration for a Virtual Node named my-node in a Mesh named my-mesh in the current AWS account and region.

# Example Updated ECS Task Role definition in CloudFormation
TaskIamRole:
    Type: AWS::IAM::Role
    Properties:
      Path: /
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service:
                - "ecs-tasks.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/CloudWatchFullAccess
        - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess
      Policies:
        - PolicyName: "AppMeshStreamAggregatedResources"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: "Allow"
                Action:
                  - "appmesh:StreamAggregatedResources"
                Resource:
                - Fn::Sub: "arn:${AWS::Partition}:appmesh:${AWS::Region}:${AWS::AccountId}:mesh/my-mesh/virtualNode/my-node"      

Option 2: Add the App Mesh Read-Only Managed Policy

The second option is to add the managed Envoy policy for AWS App Mesh, which automatically includes authorization for the appmesh:StreamAggregatedResources action on all Virtual Nodes. The managed policy will allow you to retrieve Envoy configuration for any Virtual Node in a Mesh owned by your account. If you require a narrower scope of permissions, see Option 1 above.

# Example Updated ECS Task Role definition in CloudFormation
TaskIamRole:
    Type: AWS::IAM::Role
    Properties:
      Path: /
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service:
                - "ecs-tasks.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/CloudWatchFullAccess
        - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess
        - arn:aws:iam::aws:policy/AWSAppMeshEnvoyAccess

How do I know I’ve fixed this?

You can verify you have added the proper policy changes by reviewing the instance or execution role for your compute that is running Envoy.

EC2

From the EC2 console:

1. Select your EC2 instance.
2. In the Description tab, click the link to the right of “IAM Role.”
3. In the IAM Role summary page, on the Permissions tab, you should see your custom policy for appmesh:StreamAggregatedResources, or the AWSAppMeshEnvoyAccess managed policy.

ECS with EC2 or Fargate

From the ECS console:

1. Select your ECS cluster.
2. Select your ECS service.
3. On the service detail page, click the Tasks tab.
4. Click on the Task Definition link of one of the tasks running on your service.
5. On the Task Definition detail page, click the link to the right “Task execution role.”
6. In the IAM Role summary page, on the Permissions tab, you should see your custom policy for appmesh:StreamAggregatedResources, or the AWSAppMeshEnvoyAccess managed policy.

EKS

From the EKS console:

1. Under General configuration, make note of the value under “Role ARN.”
2. Go to the IAM service in the console.
3. Click Roles on the left-hand navigation.
4. On the roles page, search for the role noted in step Region expansion #1.
5. In the IAM Role summary page, on the Permissions tab, you should see your custom policy for appmesh:StreamAggregatedResources, or the AWSAppMeshEnvoyAccess managed policy.

Once you’ve verified the appropriate role has been updated, you will be able to connect your Envoy to App Mesh.

How do I know if my Envoy is failing the authorization check?

You can determine if your Envoy is failing the authorization check by viewing the Envoy’s logs in CloudWatch. In the CloudWatch log stream, search for gRPC config stream closed: 7. If the message exists in your log stream, your Envoy has failed the App Mesh authorization check.

[bcelenza]

We've decided to extend the enforcement cut-over date from September 3rd, 2019 to November 5th, 2019.

[starchx]

Just wondering has the requirement of "appmesh:StreamAggregatedResources" been enforced yet?

I launched an ECS task without "appmesh:StreamAggregatedResources" IAM permissions on task role (and task execution role), but the task was still able to function as normal (e.g. It was still able to reach out to backend virtual services based on virtual route configured). I don't see "gRPC config stream closed: 7" in envoy cloudwatch log either. A quick confirmation will be much appreciated.

[bcelenza]

Hey @starchx, sorry we haven’t updated this issue in a bit.

The short answer is: it’s still possible to connect without the StreamAggregatedResources and receive configuration, but only a minimum configuration will be provided.

We’re enforcing the IAM action for all new features since it was introduced, which includes: TLS functionality from ACM, and Virtual Gateways.

We’ll eventually close this loop, but don’t want to break customers, so it will take us a little while to narrow down all the usages and help with migration.