k8s with VMSS for master leads to Error: invalid character ']' looking for beginning of value

[pjbgf]

Is this a request for help?:
Yes

Is this an ISSUE or FEATURE REQUEST? (choose one):
Issue

What version of acs-engine?:
v0.23.0, v0.23.1 and v0.v24.0

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
Kubernetes 1.12.1

What happened:
Error message as per below:

DEBU[0000] parsing --set flag key/value pairs from masterProfile.dnsPrefix=test
DEBU[0000] new key/value parsed: masterProfile.dnsPrefix = test
INFO[0000] new api model file has been generated during merge: C:\Users\paulo\AppData\Local\Temp\mergedApiModel052519087
WARN[0000] Clusters with VMSS masters are not yet upgradable! You will not be able to upgrade your cluster until a future version of acs-engine!
INFO[0000] Generating assets into _output/test...
DEBU[0012] pki: PKI asset creation took 7.2615164s
FATA[0012] error pretty printing template: invalid character ']' looking for beginning of value

What you expected to happen:
Either a successful deployment or generation of ARM templates.

How to reproduce it (as minimally and precisely as possible):

acs-engine.exe generate --api-model cluster-definition.json --set masterProfile.dnsPrefix="test" --debug

cluster-definition.json contains:

{
    "apiVersion": "vlabs",
    "properties": {
        "orchestratorProfile": {
            "orchestratorType": "Kubernetes",
            "orchestratorVersion": "1.12.1",
            "kubernetesConfig": {
                "useManagedIdentity": true,
                "enableEncryptionWithExternalKms": true,
                "networkPolicy": "azure",
                "enableRbac": true,
                "kubeletConfig": {
                    "--allow-privileged": "true",
                    "--anonymous-auth": "false",
                    "--max-pods": "250",
                    "--read-only-port": "0"
                },
                "gcHighThreshold":70,
                "gcLowThreshold": 60,
                "apiServerConfig": {
                    "--admission-control":  "PodSecurityPolicy,SecurityContextDeny,NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,DenyEscalatingExec,AlwaysPullImages",
                    "--request-timeout": "300s",
                    "--insecure-port": "0"
                },
                "privateCluster": {
                    "enabled": true,
                    "jumpboxProfile": {
                        "name": "vm-acs-jumpbox",
                        "vmSize": "Standard_D2_v2",
                        "osDiskSizeGB": 30,
                        "username": "someuser",
                        "publicKey": "to-be-replaced-at-deploy-time"
                      }
                },
                "addons": [
                    {
                        "name": "tiller",
                        "enabled": false
                    },
                    {
                        "name": "kubernetes-dashboard",
                        "enabled": false
                    },
                    {
                        "name": "container-monitoring",
                        "enabled": true
                    },
                    {
                        "name": "aad-pod-identity",
                        "enabled": true
                    }, 
                    {
                        "name": "keyvault-flexvolume",
                        "enabled" : true
                    }
                ]
            }
        },
        "masterProfile": {
            "count": 1,
            "vmSize": "Standard_D2_v2",
            "distro": "ubuntu",
             "availabilityProfile": "VirtualMachineScaleSets",
            "osDiskSizeGB": 50
        },
        "agentPoolProfiles": [
            {
                "name": "agentpool1",
                "count": 2,
                "vmSize": "Standard_D2s_v3",
                "distro": "ubuntu",
                "availabilityProfile": "VirtualMachineScaleSets",
                "osDiskSizeGB": 100
            }
        ],
        "linuxProfile": {
            "adminUsername": "someuser",
            "ssh": {
                "publicKeys": [
                    {
                        "keyData": "to-be-replaced-at-deploy-time"
                    }
                ]
            }
        }
    }
}

Anything else we need to know:
As soon as I remove "availabilityProfile": "VirtualMachineScaleSets" from the masterProfile, the issue is gone.

[ritazh]

Thanks for reporting this @pjbgf! Will take a look at this and get back to you.

[ritazh]

@pjbgf after some investigation we have found 3 things in this issue that need to be fixed:

1. the template generation error you see is resulted from when both enableEncryptionWithExternalKms and useManagedIdentity are enabled and master is vmss. An earlier PR caused a regression. but the issue should be fixed via Update vmss master EncryptionWithExternalKms with userassignedidentity #4082
2. "--max-pods": "250" will cause an issue with vmss instance private ip assignment. Currently default value of is 30 works as expected. Another PR will be opened to fix this.
3. privateCluster is not currently supported for vmss master. Another PR will be opened to fix this.

[ritazh]

2 and 3 have not been fixed yet.

[pjbgf]

@ritazh what exactly is the problem with the VMSS instance private IP assignment? Do you mind giving me a bit of the context, or point me to some link about the issue?

[ritazh]

@pjbgf

for 2, there is a bug/limitation in the way --max-pods is used to derive the private ip assignment of each instance in the VMSS instance. This information is used for cert generation and configs for etcd.
https://github.com/Azure/acs-engine/blob/master/pkg/api/defaults.go#L360
https://github.com/Azure/acs-engine/blob/master/parts/k8s/kubernetesmastercustomdatavmss.yml#L487-L498

for 3, private cluster support needs to be implemented and tested.

I will be looking into this shortly.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contribution. Note that acs-engine is deprecated--see https://github.com/Azure/aks-engine instead.