RFD: Automatic AWS Server Discovery

[lxea]

Related: #11628

[nklaassen]

sorry, realized a couple more things after clicking approve

[r0mant]

Makes sense in general, left a couple questions.

@klizhentas @xinding33 Could you folks take a look too from the product/UX perspective?

[xinding33]

@lxea This is a really cool feature. I find that for features related to auto discovery and enrollment, drafting user documentation helps a lot in figuring out the right UX. Can you draft some user documentation once most of the existing comments have been addressed?

[klizhentas]

I think it's a solid start. We should change the way we set this up though to support cloud use cases from the start, what rules out changing auth service configs.

Another thing to think about is that admins most likely will want to change the scripts. Having defaults is helpful, but giving them opportunity to load scripts from file and/or config section of the service going to help going forward.

[klizhentas]

Is that AWS feature?

[lxea]

Not sure what you mean here, EC2 instances can be tagged and we should be able to match on them

[klizhentas]

Where would we store this script? What if users would like to tweak it? Is that going to be a part of the SSH service config?

[lxea]

I'd intended for this to not be configurable by users, I've updated the document so the shell scripts that get run are specified in a predefined SSM document which gets executed instead of having arbitrary scripts.

This also helps to remove the restriction on the linux distros supported by this feature, which is nice

[klizhentas]

@lxea @r0mant let's also add a security section and UX section to this RFD. Let's explore attack vectors and mitigations and UX examples of how users will perceive this new functionality

[lxea]

I've moved from EC2 join tokens to IAM tokens as they dont have an expiry which makes sense in the context of this feature

The shell command running is also changed, instead of being an unconfigurable set of commands, or a shell command stored in the teleport.yaml file, a user must create a ssm command document which specifies the set of commands to run rather then sending an arbitrary shell script for each install.

I also added a UX example, what other scenarios would be helpful?

[klizhentas]

Almost there. Can we simplify the part with the instance document? Can we auto-create one if we have permissions? How can we simplify that part. Also a couple of nits on past time vs present

[nklaassen]

I am curious how the instance will prove which tags it has? The IAM join method currently only deals with an sts:GetCallerIdentity request/response, which wouldn't include this. This design involves ec2:DescribeInstances permissions on the SSH agent, but not necessary on Auth where the IAM joining verification will be run.

Which brings up a broader point: my original suggestion on this PR was to use the EC2 join method, not the IAM join method (I know they are similar and most people don't know we have both...), did you consider this? I think it would be better. Docs here in case you're unfamiliar https://goteleport.com/docs/setup/guides/joining-nodes-aws-ec2/

The EC2 join method currently requires ec2:DescribeInstances permissions on auth, but that's probably acceptable for this design. The benefits over the IAM method though are:

1. The EC2 instance does not need to have an IAM role assigned. This is a requirement for the IAM method.
2. Since auth is already calling ec2:DescribeInstances, you can probably actually enforce the tags requirement.

[nklaassen]

Hmm... sorry, I just remembered the big downside of the EC2 join method is that it doesn't work on cloud specifically because it requires the ec2:DescribeInstances permissions on auth. Not great, sorry I didn't think this through more before commenting.

However, this might be a good opportunity to allow joining with the EC2 method through an SSH agent deployed on the customer's infra, I'm not sure how this would work yet but it could be doable. Would probably require an RFD of its own.

[r0mant]

@klizhentas You requested changes here before, and @lxea has since addressed the feedback, can you take another look? There were 2 outstanding comments from what I could see:

• Teleport SSH agent attempting to automatically create SSM document. We have updated the RFD to mention this and will make appropriate update to the implementation.
• Having optional "agentless" mode for auto-discovery. We have also included an optional discovery config setting that would control this in the RFD. It is out of scope initially but we can work on this after initial release if there's still interest.

[klizhentas]

Still have a couple of questions after reading this

[klizhentas]

Is it a singleton resource? Wouldn't it be better to have multiple script names available for different instance types and specify the script per label?

[r0mant]

Right now it's a singleton, yes. Users can replace it with their own if they want to e.g. to detect instance type, etc. but if we wanted to have ability to have multiple scripts we could make this a non-singleton resource and for example add a field to auto-discovery configuration to optionally specify which install script to use.

ssh_service:
  enabled: "yes"
  aws:
   - types: ["ec2"]
     regions: ["eu-central-1"]
     tags:
       "arch": "ubuntu"
     install:
       script_name: "install-script-ubuntu"
   - types: ["ec2"]
     regions: ["eu-central-1"]
     tags:
       "arch": "centos"
     install:
       script_name: "install-script-centos"

This would require users to mark their AWS nodes with appropriate tags if they want to use different install scripts. Auto-discovery configuration will become dynamic as a part of Teleport Discover Day-2 work so they will be able to update this mapping dynamically as well when we implement it.

@klizhentas @lxea What do you think?

[lxea]

This seems good to me, would the script_name option here just be used for parameterize the SSM command run?

[klizhentas]

@lxea yes

[klizhentas]

How do I specify what SSM command and script to run? Would it just run one script for everything? Can we add a script doc name as an optional parameter here? We should create a default one, but it can't be singleton

[r0mant]

Yes, it works like you describe. We create a default SSM document but you can optionally specify a different one to use. There's a configuration example somewhere above:

  aws:
   - types: ["ec2"]
     regions: ["eu-central-1"]
     tags:
       "teleport": "yes"
     ssm:
       document: "TeleportDiscoveryInstaller" # default value

@lxea Please correct me if I'm wrong.

[lxea]

this is correct, responded to comments about different installer scripts above

[r0mant]

@klizhentas We have converted the installer resource from singleton to multiple resources (#15682). Could you take another look and let us know if this lgty now?