[ECS][Fargate]: Add EFS volumes through CloudFormation

[srrengar]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
CloudFormation support for adding EFS volumes to ECS tasks on Fargate or EC2.

Which service(s) is this request for?
Fargate, ECS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
EFS support launched for ECS on Fargate and EC2. Customers deploying through CloudFormation should also be able to use this feature.

[rpgreen]

Is there a timeline on this? It is disappointing to see feature releases without CFN support on day 1. We will have to jump through some hoops to make EFS work with our current setup.

[gary-cowell]

And I need it to work through the CDK. Which requires cloudformation first.

[guillaumesmo]

Please note you can use custom resources while a feature is not yet supported by CloudFormation

I wrote a simple proof of concept to create an ECS task definition with EFS mount through CloudFormation:
https://gist.github.com/guillaumesmo/4782e26500a3ac768888daab3c55b139

[Trandel]

👍

[jedis00]

Any update on this one?

[machielg]

I'm using a custom resource but the sad thing is that I need to replicate the whole task definition creation in stead of just amending the volume part. This issue is also blocking the inclusion into CDK. aws/aws-cdk#6918

[skysb]

Hi guys, is there an update on when the cloudformation would be updated with the EFS inclusion for ECS?

[srrengar]

Hi everyone, thank you for the valuable feedback. I can assure you that this is coming soon and we will provide updates here when we have more to share. We decided to release EFS for ECS (on both EC2 and Fargate) sooner so that customers can use it and provide early feedback instead of waiting till CloudFormation was ready. We understand CloudFormation is a great way to create and update resources to use new features, and we are working on improving how we release CloudFormation support for new features going forward.

[cajames]

if anyone's still looking for a way to do this in CDK, here's a sample gist with a workaround, going down the path @guillaumesmo mentions above: https://gist.github.com/cajames/3daec680b1101c8358e2ff30dfadd52a

The key takeaway, creating a custom resource Task Definition, and then using CDK "Raw Overrides" to attach it to the service. Documented here: https://docs.aws.amazon.com/cdk/latest/guide/cfn_layer.html

I've explained the method here: aws/aws-cdk#6240 (comment)

Hope this can help anyone. This is a temporary workaround, and once this issue is resolved, it'll be an easier switch to remove the custom task definition and add the config directly into the service. :)

[bellaxievmlyr]

Any update on how long it would take? A few days or weeks or months? I'm currently blocked by this issue. If it's a few days, I might just wait rather than spending a few days to figure out a workaround.

[machielg]

Any update on how long it would take? A few days or weeks or months? I'm currently blocked by this issue. If it's a few days, I might just wait rather than spending a few days to figure out a workaround.

It moved from ‘in progress’ to ‘coming soon’ which also includes issues from 2019. Not a good sign 🤔

[rpgreen]

Folks, it's been over 3 months since this feature was launched and there's STILL no CFN support. This will definitely make me think twice about adopting new ECS features in the future. CFN support is not a 'nice to have', it should be included on launch.

[Samseppiol]

Folks, it's been over 3 months since this feature was launched and there's STILL no CFN support. This will definitely make me think twice about adopting new ECS features in the future. CFN support is not a 'nice to have', it should be included on launch.

Are we thinking about the same company? This is AWS, cfn support isn't for atleast 12-18 months after announcing clickops.

[sandeepboyapati]

eagerly waiting

[bushong1]

It's 2020. CloudFormation has been out for 9 years. How is this still happening?

[ghferrari]

I'm eagerly looking forward to this, too and I hope we'll be kept regularly updated on progress.

Just to give a bit of feedback to @srrengar , in my view there is simply no such thing as "releasing a feature" without cloudformation support. As far as I'm concerned, either it's in cloudformation where it can be documented, placed in version control and reliably, repeatedly tested and deployed or it doesn't exist. As a sole developer, I don't have time for manual features and wouldn't even consider using them.

So this is me giving some feedback and a big show of encouragement to the idea of "reducing the time between a feature being released and getting cloudformation support". But I'd really like you to go all the way and understand that for many of us, a new feature is only really released when it becomes available in cloudformation.

[craighurley]

All these small straws triggered my move to terraform. So far so good, plus I get to skill up on a tool that can deploy to other cloud providers.

[gary-cowell]

Just to give a bit of feedback to @srrengar , in my view there is simply no such thing as "releasing a feature" without cloudformation support. As far as I'm concerned, either it's in cloudformation where it can be documented, placed in version control and reliably, repeatedly tested and deployed or it doesn't exist. As a sole developer, I don't have time for manual features and wouldn't even consider using them.

So this is me giving some feedback and a big show of encouragement to the idea of "reducing the time between a feature being released and getting cloudformation support". But I'd really like you to go all the way and understand that for many of us, a new feature is only really released when it becomes available in cloudformation.

Even worse, is, I have the same goals of documented code in version control and repeatable reliable testing, but , I need this from the AWS CDK. As CDK relies on cloudformation, I have no doubt that any CDK release of this will be well behind any eventual cloudformation update.

[ I know you can mess with the cloudformation in CDK even without full support - I'd rather not do that ]

[mwarkentin]

Looks like this is supported now: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-taskdefinition.html#cfn-ecs-taskdefinition-volumes

[GeoffWilliams]

it works! I just hit the outdated lambda SDK issue where it rejects valid task definitions that use access points so now I can use this 🍾

[frank69m]

So are you guys saying that with CDK 1.56, I can now mount EFS volumes in my task definition.... I glanced at the docs, not many good examples

[frank69m]

Ok. So I just looked at the docs. I'm looking at mounting an EFS volume to a Fargate task definition with the cdk

It always seems to revert to Bind Mount

in my TD for python

volume=[
{
"name": "EFS",
"efsVolumeConfiguration": {
"fileSystemId": mount.file_system_id,
"rootDirectory": "/"
}
}
]

It does a synth fine, but when I checkout the TD, it creates the name, but it's still stuck at Bind Mount

any ideas? Please help

[srrengar]

hey everyone, this is supported now. The official announcement will be published Monday

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-efsvolumeconfiguration.html

[ryanchapman]

Since it took me a few hours to glue CFN resources together to get this working, thought I'd post it here. Hopefully it saves others some time.

If you run into unsuccessful EFS utils command execution; code: 32, then you may have security group issues. Check the ingress on your sg associated with your mount targets (GrafanaEFSServerSecurityGroup in this example) as well as egress on the sg associated with your ECS service (GrafanaTaskSecurityGroup in this example).

AWSTemplateFormatVersion: 2010-09-09
[...]
Resources:
[...]
  GrafanaEFSVolume:
    Type: AWS::EFS::FileSystem
    Properties:
      Encrypted: true
      FileSystemTags:
        - Key: Name
          Value: !Sub "${AWS::StackName}-efs-vol"

  GrafanaEFSServerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub "${AWS::StackName}-efs-server-endpoints"
      GroupDescription: Which client ip addrs are allowed to access EFS server endpoints for grafana mount
      VpcId:
        Fn::ImportValue: !Sub "${VPCStack}-VPC"
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 2049
          ToPort: 2049
          CidrIp:
            Fn::ImportValue: !Sub "${VPCStack}-CIDR"
      Tags:
        - Key: Name
          Value: !Sub "${AWS::StackName}-efs-server"

  GrafanaEFSMountTarget0:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref GrafanaEFSVolume
      SecurityGroups:
        - !Ref GrafanaEFSServerSecurityGroup
      SubnetId:
        Fn::Select:
          - 0
          - Fn::Split:
            - ','
            - Fn::ImportValue: !Sub "${VPCStack}-PrivateSubnets"

  GrafanaEFSMountTarget1:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref GrafanaEFSVolume
      SecurityGroups:
        - !Ref GrafanaEFSServerSecurityGroup
      SubnetId:
        Fn::Select:
          - 1
          - Fn::Split:
            - ','
            - Fn::ImportValue: !Sub "${VPCStack}-PrivateSubnets"

  GrafanaEFSMountTarget2:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref GrafanaEFSVolume
      SecurityGroups:
        - !Ref GrafanaEFSServerSecurityGroup
      SubnetId:
        Fn::Select:
          - 2
          - Fn::Split:
            - ','
            - Fn::ImportValue: !Sub "${VPCStack}-PrivateSubnets"

  GrafanaTaskDefinition:
    Type: AWS::ECS::TaskDefinition
    DependsOn:
      - ECSExecutionPolicy
    Properties:
      Volumes:
        - Name: grafana
          EFSVolumeConfiguration:
            FilesystemId: !Ref GrafanaEFSVolume
            TransitEncryption: ENABLED
      ContainerDefinitions:
        - Name: grafana
          Image: !Sub "${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${GrafanaImage}"
          # Running as root to set perms on EFS mount before running grafana
          User: root
          Entrypoint: ['bash', '-c']
          Command:
            - |
              chown -R grafana:grafana /var/lib/grafana
              su grafana -s /bin/bash -c /run.sh
          MountPoints:
            - SourceVolume: grafana
              ContainerPath: '/var/lib/grafana'
              ReadOnly: false
          StopTimeout: 60
          PortMappings:
            - ContainerPort: 3000
              HostPort: 3000
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-region: !Ref 'AWS::Region'
              awslogs-group: !Ref ECSLogGroup
              awslogs-stream-prefix: ecs-grafana
      Cpu: '512'
      Memory: '2048'
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      ExecutionRoleArn: !GetAtt ECSExecutionRole.Arn
      TaskRoleArn: !GetAtt ECSTaskRole.Arn

  GrafanaTaskSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub "${AWS::StackName}-grafana-task"
      GroupDescription: Security group for the grafana task.
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 3000
          ToPort: 3000
          SourceSecurityGroupId:
            Fn::ImportValue: !Sub "${PlatformStack}-InternalLoadBalancerServerSecurityGroup"
      SecurityGroupEgress:
        - IpProtocol: tcp
          FromPort: 2049
          ToPort: 2049
          CidrIp:
            Fn::ImportValue: !Sub "${VPCStack}-CIDR"
      VpcId:
        Fn::ImportValue: !Sub "${VPCStack}-VPC"

  GrafanaService:
    Type: AWS::ECS::Service
    Properties:
      Cluster:
        Fn::ImportValue: !Sub "${PlatformStack}-ECSCluster"
      DesiredCount: 1
      DeploymentConfiguration:
        MaximumPercent: 200
        MinimumHealthyPercent: 100
      LaunchType: FARGATE
      LoadBalancers:
        - ContainerName: grafana
          ContainerPort: 3000
          TargetGroupArn: !Ref InternalLoadBalancerTargetGroup
      NetworkConfiguration:
        AwsvpcConfiguration:
          SecurityGroups:
            - Fn::ImportValue: !Sub "${PlatformStack}-InternalLoadBalancerClientSecurityGroup"
            - !Ref GrafanaTaskSecurityGroup
          Subnets:
            Fn::Split:
              - ','
              - Fn::ImportValue: !Sub "${VPCStack}-PrivateSubnets"
      PlatformVersion: '1.4.0'
      TaskDefinition: !Ref GrafanaTaskDefinition

[frank69m]

Has anyone confirmed if this works with the CDK.

Everytime I synth and try to look at my CF, it only shows

"Volumes": [
{
"Name": "testefs"
}
]

And yes, I did put in the efsVolumeConfiguration

[engineal]

Has anyone confirmed if this works with the CDK.

Everytime I synth and try to look at my CF, it only shows

"Volumes": [
{
"Name": "testefs"
}
]

And yes, I did put in the efsVolumeConfiguration

@frank69m While I don't think this feature has been added to the CDK generated CFN resources or higher level constructs yet, I was able to use this feature through the CDK escape hatch: https://docs.aws.amazon.com/cdk/latest/guide/cfn_layer.html

To my Fargate task definition I added a property override:

const cfnTask = this.task.node.defaultChild as CfnTaskDefinition;
cfnTask.addPropertyOverride('Volumes', [{
    EFSVolumeConfiguration: {
        FilesystemId: this.fileSystem.fileSystemId,
        TransitEncryption: 'ENABLED'
    },
    Name: 'efs'
}]);

and to the container added to that task definition I added a mount point:

container.addMountPoints({
    sourceVolume: 'efs',
    containerPath: '/data',
    readOnly: false
});

I have successfully used this to add an EFS mount to a Fargate task.

[cmoreno82]

I still have only the "docker_volume_configuration" option with CDK 1.56.0 and python. I use the ecs.FargateTaskDefinition.add_volume function:

Task Definition

      fargate_task_definition_zk = ecs.FargateTaskDefinition(
          self, "task_" + zk_name,
          family="task-" + zk_name,
          execution_role=self.role,
          memory_limit_mib=self.node.try_get_context("zookeeper_memory"),
          cpu=self.node.try_get_context("zookeeper_cpu")
      )     
       fargate_task_definition_zk.add_volume(
          name="test",
          EFSVolume....  <--- Only "docker_volume_configuration" is possible
      )

Someone else test it with cdk+python?

[srrengar]

Here is the official announcement - https://aws.amazon.com/about-aws/whats-new/2020/08/amazon-ecs-announces-cloudformation-support-for-amazon-efs-volumes/

[frank69m]

great. works now with CDK and the @engineal solution above.

[paveljos]

Just to close the gap for anyone looking for a python solution based on @engineal 's submission, here's what I implemented in CDK:

cfn_task = self.task_definition.node.default_child
cfn_task.add_property_override("Volumes", [{
   "EFSVolumeConfiguration": {
      "FilesystemId": efs_id.file_system_id,
      "TransitEncryption": "ENABLED"
   },
   "Name": "efs"
}])

[skindc]

For anyone still having issues with this, I seems ensuring that you explicitly set the platform version in your containers definition to '1.4.0' solves the problem. When using 'latest' this is actually implying 1.3.0 which seems does not support the latest announcements.

[yogeshjoshi]

For anyone still having issues with this, I seems ensuring that you explicitly set the platform version in your containers definition to '1.4.0' solves the problem. When using 'latest' this is actually implying 1.3.0 which seems does not support the latest announcements.

Yes @skindc , yes this has been solved in version 1.4.0 in August this year.
Latest version in definition refers 1.3.0

[mreferre]

@skindc / @yogeshjoshi tangential to this thread, we have an open issue around how to better manage/communicate new Platform Versions. If you have opinions or feedback please leave a comment at #1069 . Thanks!

[eob]

@engineal I'm using your solution above, but when using ecs_patterns.ApplicationLoadBalancedFargateService with the taskImageOptions I can't figure out how to access my container object, as in:

container.addMountPoints({
    sourceVolume: 'efs',
    containerPath: '/data',
    readOnly: false
});

Do you know if it's still possible to attach an EFS volume to a container that has been created with:

      taskImageOptions: { 
        image: ContainerImage.fromEcrRepository(...),
        environment: { ... },
        containerPort: 8080,
        enableLogging: true,
      },

[engineal]

Hi @eob and anyone else who is interested. I believe the property override I had suggested before isn't required anymore, as these props are now supported by the aws-cdk constructs.

I can replace my previous use of the property override with a call to the addVolume method on my task definition:

this.task.addVolume({
    name: "efs",
    efsVolumeConfiguration: {
        fileSystemId: this.fileSystem.fileSystemId,
        transitEncryption: "ENABLED"
    }
});

And just like before, to the container added to that task definition I added a mount point referencing that volume:

container.addMountPoints({
    sourceVolume: 'efs',
    containerPath: '/data',
    readOnly: false
});

Now for your example @eob, if you've created your service using the ecs-patterns ApplicationLoadBalancedFargateService construct like so:

const service = new ApplicationLoadBalancedFargateService(this, 'service', {
    taskImageOptions: {
        image: ContainerImage.fromEcrRepository(...),
        environment: { ... },
        containerPort: 8080,
        enableLogging: true,
    }
});

you can access the task definition created by that service using service.taskDefinition, and add the EFS volume to the task definition like so:

service.taskDefinition.addVolume({
    name: "efs",
    efsVolumeConfiguration: {
        fileSystemId: this.fileSystem.fileSystemId,
        transitEncryption: "ENABLED"
    }
});

Now the ApplicationLoadBalancedFargateService construct adds a single container to the task definition, which will either be named 'web' or the value you assign containerName in your taskImageOptions. This means you can access that container by using taskDefinition.findContainer('web') (or taskDefinition.findContainer(containerName) if you specified containerName in your taskImageOptions). Additionally, since the container added by the ApplicationLoadBalancedFargateService is the first container added, you can access it by using taskDefinition.defaultContainer as well.

So, to add the mount point to the container created by the ApplicationLoadBalancedFargateService, you can do:

service.taskDefinition.findContainer('web')?.addMountPoints({
    sourceVolume: 'efs',
    containerPath: '/data',
    readOnly: false
});

or

service.taskDefinition.defaultContainer?.addMountPoints({
    sourceVolume: 'efs',
    containerPath: '/data',
    readOnly: false
});

[eob]

@engineal thank you for so kindly explaining that!

Your comment got everything deploying properly, but I'm still getting permissions errors when I try to actually access the volume.

I've seen a few articles about configuring AccessPoints with EFS, but most Fargate tutorials proceed without them as if they're not actually necessary. Do you know if there any permissions gotchas that I might not be noticing?

This is what my code looks like -- I copy-paste-customized the ApplicationLoadBalancedFargateService code so that it's easier to just pass in mounts as an option.

// FILESYSTEM
const fs = new efs.FileSystem(this, "IndexFilesystem", {
  vpc: vpc,
  performanceMode: efs.PerformanceMode.GENERAL_PURPOSE,
  throughputMode: efs.ThroughputMode.BURSTING,
  fileSystemName: "indexFilesystem"
})
  
// VOLUME
const volume = {      
  name: "indexVolume",
  efsVolumeConfiguration: {
    fileSystemId: fs.fileSystemId,
    TransitEncryption: 'ENABLED',
  },
};

// this is a FargateTaskDefinition object
this.taskDefinition.addVolume(volume)  


// MOUNT POINT
const mountPoint = {
  containerPath: "/data",
  sourceVolume: volume.name,
  readOnly: false,
};

// This is the container created inside ApplicationLoadBalancedFargateService
container.addMountPoints(mountPoint)

// PROVIDE ACCESS
// this.service is the FargateService
this.service.connections.allowFrom(fs, Port.tcp(2049));
this.service.connections.allowTo(fs, Port.tcp(2049));

[engineal]

@engineal thank you for so kindly explaining that!

Your comment got everything deploying properly, but I'm still getting permissions errors when I try to actually access the volume.

I've seen a few articles about configuring AccessPoints with EFS, but most Fargate tutorials proceed without them as if they're not actually necessary. Do you know if there any permissions gotchas that I might not be noticing?

This is what my code looks like -- I copy-paste-customized the ApplicationLoadBalancedFargateService code so that it's easier to just pass in mounts as an option.

@eob Would you happen to be able to describe or include in a comment the error message that you receive?

[sukrit007]

@eob What is the user (UID/GID) of the user running within the container? You might have to create an access point for specific Posix user to get around Posix permissions. https://docs.aws.amazon.com/efs/latest/ug/accessing-fs-nfs-permissions.html

[fire015]

Just adding to what @eob and @sukrit007 said, if you want to use an access point in your Fargate service so that you can mount as non-root here is what I did (assume 2000 is the POSIX UID that runs inside the container):

const fs = new efs.FileSystem(this, "Filesystem", {
  vpc: vpc,
  performanceMode: efs.PerformanceMode.GENERAL_PURPOSE,
  throughputMode: efs.ThroughputMode.BURSTING,
  fileSystemName: "MyFilesystem",
});

const accessPoint = new efs.AccessPoint(this, "AccessPoint", {
  fileSystem: fs,
  path: "/data",
  createAcl: {
    ownerUid: "2000",
    ownerGid: "2000",
    permissions: "0755",
  },
  posixUser: {
    uid: "2000",
    gid: "2000",
  },
});

const volume: ecs.Volume = {
  name: "MyVolume",
  efsVolumeConfiguration: {
    fileSystemId: fs.fileSystemId,
    transitEncryption: "ENABLED",
    authorizationConfig: {
      accessPointId: accessPoint.accessPointId,
    },
  },
};

fargateService.taskDefinition.addVolume(volume);

fargateService.taskDefinition.defaultContainer!.addMountPoints({
  containerPath: "/data",
  sourceVolume: volume.name,
  readOnly: false,
});

fargateService.service.connections.allowFrom(fs, ec2.Port.tcp(efs.FileSystem.DEFAULT_PORT));
fargateService.service.connections.allowTo(fs, ec2.Port.tcp(efs.FileSystem.DEFAULT_PORT));