Add support for SpotPrice and Mixed Instance ASGs

[rifelpet]

fixes #7049

I initially added SpotPrice to the LaunchTemplate directly like this:

https://github.com/rifelpet/kops/blob/a83a178dad892da9d194764fb1ea91eba70c51bb/upup/pkg/fi/cloudup/awstasks/launchtemplate_target_api.go#L120-L127

but that resulted in the following error from AWS:

W0524 15:36:30.395412   15535 executor.go:130] error running task "AutoscalingGroup/test.foobar" (9m56s remaining to succeed): error creating AutoscalingGroup: InvalidQueryParameter: Incompatible launch template: You cannot use a launch template that is set to request Spot Instances (InstanceMarketOptions) when you configure an Auto Scaling group with a mixed instances policy. Add a different launch template to the group and try again.
	status code: 400, request id: 5e45cf09-7e74-11e9-9172-b737eb317e73

So instead it needs to be in the ASG's MixedInstancesPolicy.InstancesDistribution.

This now correctly adds the spot price to the mixed instances ASG:

Notes:

• I had to make the LaunchTemplate task's SpotPrice optional because we weren't setting it in AWS when using mixed instances which resulted in these erroneous changes being reported on subsequent cluster updates:

Will modify resources:
  LaunchTemplate/test.foobar
  	SpotPrice           	  -> 0.01

• I had to change the SpotMaxPrice's type in Terraform and CloudFormation outputs to string, but it didn't appear to actually get used anywhere so I think thats ok. It is indeed a string in the Terraform provider:

https://github.com/terraform-providers/terraform-provider-aws/blob/240d623a35b18f8a83b0803c8d5b0e3a7f273547/aws/resource_aws_launch_template.go#L289-L292

While adding integration tests I discovered another bug in which Kops was not adding node_autoscaling_group_ids, node_security_group_ids, and node_subnet_ids terraform outputs when using launch templates. This fixes that as well.

Without the two fixes, the new integration tests fails with:

--- FAIL: TestMixedInstancesSpotASG (1.32s)
    /gopath/src/k8s.io/kops/cmd/kops/integration_test.go:312: diff:
        ...
            masters_role_arn             = "${aws_iam_role.masters-mixedinstances-example-com.arn}"
            masters_role_name            = "${aws_iam_role.masters-mixedinstances-example-com.name}"
        -   node_autoscaling_group_ids   = ["${aws_autoscaling_group.nodes-mixedinstances-example-com.id}"]
        -   node_security_group_ids      = ["${aws_security_group.nodes-mixedinstances-example-com.id}"]
        -   node_subnet_ids              = ["${aws_subnet.us-test-1b-mixedinstances-example-com.id}"]
            nodes_role_arn               = "${aws_iam_role.nodes-mixedinstances-example-com.arn}"
            nodes_role_name              = "${aws_iam_role.nodes-mixedinstances-example-com.name}"
        ...
          }
          
        - output "node_autoscaling_group_ids" {
        -   value = ["${aws_autoscaling_group.nodes-mixedinstances-example-com.id}"]
        - }
        - 
        - output "node_security_group_ids" {
        -   value = ["${aws_security_group.nodes-mixedinstances-example-com.id}"]
        - }
        - 
        - output "node_subnet_ids" {
        -   value = ["${aws_subnet.us-test-1b-mixedinstances-example-com.id}"]
        - }
        - 
          output "nodes_role_arn" {
            value = "${aws_iam_role.nodes-mixedinstances-example-com.arn}"
        ...
                on_demand_percentage_above_base_capacity = 5
                spot_instance_pools                      = 3
        -       spot_max_price                           = "0.1"
              }
            }
        ...
            key_name      = "${aws_key_pair.kubernetes-mixedinstances-example-com-c4a6ed9aa889b9e2c39cd663eb9c7157.id}"
          
        +   instance_market_options = {
        +     market_type = "spot"
        + 
        +     spot_options = {
        +       max_price = "0.1"
        +     }
        +   }
        + 
            network_interfaces = {
              associate_public_ip_address = true
        ...
        
    /gopath/src/k8s.io/kops/cmd/kops/integration_test.go:323: terraform output differed from expected

[rifelpet]

/retest

[rifelpet]

/test pull-kops-bazel-test

[rdrgmnzs]

Looks great @rifelpet, thanks!

/assign @gambol99 @granular-ryanbonham

[rifelpet]

I rebased and added two new integration tests, one using only mixed instances and one using mixed instances and maxPrice. To aid in review, heres the diff between the two test directories:

diff -c -r tests/integration/update_cluster/mixed_instances/cloudformation.json tests/integration/update_cluster/mixed_instances_spot/cloudformation.json
*** tests/integration/update_cluster/mixed_instances/cloudformation.json        2019-06-07 14:25:09.000000000 -0700
--- tests/integration/update_cluster/mixed_instances_spot/cloudformation.json   2019-06-07 14:18:48.000000000 -0700
***************
*** 206,212 ****
            },
            "InstancesDistribution": {
              "OnDemandPercentageAboveBaseCapacity": 5,
!             "SpotInstancePool": 3
            }
          }
        }
--- 206,213 ----
            },
            "InstancesDistribution": {
              "OnDemandPercentageAboveBaseCapacity": 5,
!             "SpotInstancePool": 3,
!             "SpotMaxPrice": "0.1"
            }
          }
        }
diff -c -r tests/integration/update_cluster/mixed_instances/in-v1alpha2.yaml tests/integration/update_cluster/mixed_instances_spot/in-v1alpha2.yaml
*** tests/integration/update_cluster/mixed_instances/in-v1alpha2.yaml   2019-06-07 14:29:34.000000000 -0700
--- tests/integration/update_cluster/mixed_instances_spot/in-v1alpha2.yaml      2019-06-07 14:16:54.000000000 -0700
***************
*** 70,75 ****
--- 70,76 ----
    role: Node
    subnets:
    - us-test-1b
+   maxPrice: "0.1"
    mixedInstancesPolicy:
      instances:
      - m5.large
diff -c -r tests/integration/update_cluster/mixed_instances/kubernetes.tf tests/integration/update_cluster/mixed_instances_spot/kubernetes.tf
*** tests/integration/update_cluster/mixed_instances/kubernetes.tf      2019-06-07 14:16:54.000000000 -0700
--- tests/integration/update_cluster/mixed_instances_spot/kubernetes.tf 2019-06-07 14:18:48.000000000 -0700
***************
*** 205,210 ****
--- 205,211 ----
      instances_distribution = {
        on_demand_percentage_above_base_capacity = 5
        spot_instance_pools                      = 3
+       spot_max_price                           = "0.1"
      }
    }
 

[mikesplain]

Great fix @rifelpet! Thanks for the contribution, as always!

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mikesplain, rifelpet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mikesplain]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rifelpet]

@mikesplain I was planning on cherry-picking this to 1.14 and 1.13. Do you think it should be in 1.12 as well?