Knative on AKS docs: istio installation istio-statsd-prom-bridge crashing

[chanezon]

Expected Behavior

Following https://github.com/knative/docs/blob/master/install/Knative-with-AKS.md

kubectl get pods -n istio-system

should yield a set of running or completed pods.

Actual Behavior

Instead, pod istio-statsd-prom-bridge is in CrashLoopBackOff state.

NAME                                       READY     STATUS             RESTARTS   AGE
istio-citadel-7bdc7775c7-cc6fv             1/1       Running            0          3d
istio-cleanup-old-ca-g7h8b                 0/1       Completed          0          3d
istio-egressgateway-795fc9b47-jd5vv        1/1       Running            0          3d
istio-ingress-84659cf44c-clmgn             1/1       Running            0          3d
istio-ingressgateway-7d89dbf85f-2x4vn      1/1       Running            0          3d
istio-mixer-post-install-jbfs5             0/1       Completed          0          3d
istio-pilot-66f4dd866c-gvblb               2/2       Running            31         3d
istio-policy-76c8896799-x9tq8              2/2       Running            0          3d
istio-sidecar-injector-645c89bc64-9b2b9    1/1       Running            0          3d
istio-statsd-prom-bridge-949999c4c-s55bq   0/1       CrashLoopBackOff   1088       3d
istio-telemetry-6554768879-plnpk           2/2       Running            0          3d
knative-ingressgateway-5f5dc4b4cd-p4dlr    1/1       Running            0          3d
zipkin-795dcbf84b-8thbv                    1/1       Running            0          3d

Steps to Reproduce the Problem

Follow the steps in
https://github.com/knative/docs/blob/master/install/Knative-with-AKS.md until https://github.com/knative/docs/blob/master/install/Knative-with-AKS.md#installing-knative-components
1.
2.
3.

Additional Info

[knative-prow-robot]

@chanezon: GitHub didn't allow me to assign the following users: user.

Note that only knative members and repo collaborators can be assigned.
For more information please see the contributor guide

In response to this:

Expected Behavior

Following https://github.com/knative/docs/blob/master/install/Knative-with-AKS.md

kubectl get pods -n istio-system

should yield a set of running or completed pods.

Actual Behavior

Instead, pod istio-statsd-prom-bridge is in CrashLoopBackOff state.

NAME                                       READY     STATUS             RESTARTS   AGE
istio-citadel-7bdc7775c7-cc6fv             1/1       Running            0          3d
istio-cleanup-old-ca-g7h8b                 0/1       Completed          0          3d
istio-egressgateway-795fc9b47-jd5vv        1/1       Running            0          3d
istio-ingress-84659cf44c-clmgn             1/1       Running            0          3d
istio-ingressgateway-7d89dbf85f-2x4vn      1/1       Running            0          3d
istio-mixer-post-install-jbfs5             0/1       Completed          0          3d
istio-pilot-66f4dd866c-gvblb               2/2       Running            31         3d
istio-policy-76c8896799-x9tq8              2/2       Running            0          3d
istio-sidecar-injector-645c89bc64-9b2b9    1/1       Running            0          3d
istio-statsd-prom-bridge-949999c4c-s55bq   0/1       CrashLoopBackOff   1088       3d
istio-telemetry-6554768879-plnpk           2/2       Running            0          3d
knative-ingressgateway-5f5dc4b4cd-p4dlr    1/1       Running            0          3d
zipkin-795dcbf84b-8thbv                    1/1       Running            0          3d

Steps to Reproduce the Problem

Follow the steps in
https://github.com/knative/docs/blob/master/install/Knative-with-AKS.md until https://github.com/knative/docs/blob/master/install/Knative-with-AKS.md#installing-knative-components
1.
2.
3.

Additional Info

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tcnghia]

This is also reported in knative/serving#1954 and knative/serving#1921 .

It is being fixed upstream in istio/istio#8249 .

@chanezon is this a blocking issue when using Knative 0.1?

[chanezon]

I don't know if it is blocking for all scenarios: I am running into issues in source-to-url sample
knative/build#313
and noticed that istio install was not clean, so thought maybe it has an impact.
I'll try with istio 1.0

[chanezon]

I used the fix @sdake provided in knative/serving#1921, ie going back to statsd 0.6.0, it solved the istio issue, but not the source-to-url issue.

A lot of "failed to list" "failed to watch" error messages in the knative-controller logs

E0827 21:33:18.527911       1 reflector.go:205] github.com/knative/serving/pkg/client/informers/externalversions/factory.go:72: Failed to list *v1alpha1.Route: the server was unable to return a response in the time allotted, but may still be processing the request (get routes.serving.knative.dev)
E0827 21:33:32.448591       1 reflector.go:322] github.com/knative/serving/pkg/client/informers/externalversions/factory.go:72: Failed to watch *v1alpha1.Service: the server was unable to return a response in the time allotted, but may still be processing the request (get services.serving.knative.dev)

[sdake]

@chanezon the failure to watch is a limitation at this time of the AKS platform as far as we can tell. With knative, you are likely running into the same limitation. We speculate AKS kube apiserver doesn't have sufficient resources to process significant (20+) watchers. See here for more details:

istio/istio#7675

with a response from AKS support:
istio/istio#7675 (comment)

and a response from a Microsoft upstream developer:
istio/istio#7675 (comment)

Cheers
-steve

[sdake]

And Microsoft's response: Azure/AKS#620

Cheers
-steve

[chanezon]

Thank you @sdake, I'll play with Knative elsewhere until these issues are sorted out on AKS.

[cforce]

Btw. it also happens if you play that on docker for windows edge (statsd_exporter: error: unknown short flag '-s', try --help) . When using https://github.com/knative/serving/blob/master/third_party/istio-1.0.2/istio.yaml issue is gone