Slave-only spot instances

[nchammas]

Flintrock supports launching clusters using spot instances. However, when doing so, all the cluster instances are launched as spot instances.

Generally, though, you don't want the master to be a spot instance. You want it to stick around even if various slaves are dropped so that your job can still chug along (though at a slower pace) and so that you can re-add slaves to the cluster at a later time.

We should change our support for launching clusters on spot instances such that only the slaves are spot instances.

This work (kinda) depends on #16.

[engrean]

@nchammas would this solve the problem that occurs when launching a spot instance cluster and one of the nodes ends up getting terminated which ends in the launch command just hanging forever? Or should I submit a new issue for that case?

[nchammas]

@engrean - Have you seen this issue since d5b086c made it in? Flintrock should now immediately error out as soon as a spot request fails.

You'll need to be running on master to get this fix.

If you're still seeing the issue, the yes, please submit a new issue so we can look into it. I'd be curious to see where exactly the launch hangs, for example.

[sylvinus]

Would love to have this feature. It would be great to be able to specify a different instance type for the master as well.

[ktdrv]

Would it make sense to separate the add-slaves logic from launch? In other words, the workflow would be to create a master instance only first and then add all the slaves of any type you want to it with a second call.

[DBCerigo]

Seconding the "would love this feature", and even more so, being able to specify different master and slave instance types.

[mblackgeo]

@nchammas is there any progress on this?

[nchammas]

@mblack20 - Nope. The current workaround, I believe, is to launch a regular cluster with no slaves, and then add spot slaves separately using add-slaves.