SPARK-2246: Add user-data option to EC2 scripts

[douglaz]

No description provided.

[AmplabJenkins]

Can one of the admins verify this patch?

[AmplabJenkins]

Can one of the admins verify this patch?

[pwendell]

Hey @douglaz - I looked around at the AWS API and it appears that this is a somewhat generic API (i.e. it doesn't necessarily have to contain a script).

UserData - string - Optional - Optional data, specific to a user’s application, to provide in the launch request. All instances that collectively comprise the launch request have access to this data. User data is never returned through API responses.

http://docs.aws.amazon.com/AWSSDKforPHP/latest/index.html#m=AmazonEC2/request_spot_instances

I couldn't find much more information than this online. Is this something where a few specific AMI types interpret this as a script and execute it on start-up? If that's the case maybe we should have a more generic description of the option, since the current description implies this will always be treated as a script

[douglaz]

@pwendell I think all AMIs interpret this way (at least all based on Amazon Linux or Ubuntu, from my experience). If it starts with a hashbang, then it is executed on start-up.

Yes, it can be anything (and the code accepts anything), but I've seen it only being used as some sort of script.

Feel free to change the description. I described this way because it's a good combination between being short, generic and informative. Otherwise I would describe it as "The EC2 data stuff. Consult the Amazon documentation for more (poor) information"