No services available through Jaeger on Istio 0.2.4

[redhotpenguin]

After having setup Jaeger via, https://istio.io/docs/tasks/telemetry/distributed-tracing.html, the dashboard is accessible but the services select box only contains a -.

Environment - macOS Sierra. Versions of relevant brew packages are:

docker 17.06.2-ce-mac27,19124
java 1.8.0_144-b01,090f390dda5b47b9b721c7dfaa008135
minikube 0.22.2
vagrant 1.7.4
virtualbox 5.1.28-117968

jaeger-deployment pod is running:

NAME                                 READY     STATUS             RESTARTS   AGE
details-v1-2202528699-88cjd          2/2       Running            0          1h
grafana-1916556110-bs35j             1/1       Running            0          1h
istio-egress-664558847-8lsbz         1/1       Running            18         1h
istio-ingress-1292229662-sl67j       0/1       CrashLoopBackOff   15         1h
istio-mixer-2555573127-kgqkc         1/1       Running            0          1h
istio-pilot-4143248751-jj63n         2/2       Running            1          1h
jaeger-deployment-1058555976-k32h0   1/1       Running            0          19m
productpage-v1-849198989-kpc87       2/2       Running            0          1h
prometheus-4245872192-w0dvg          1/1       Running            0          1h
ratings-v1-217391091-6m7p2           2/2       Running            0          1h
reviews-v1-3397570414-f15xf          2/2       Running            0          1h
reviews-v2-755907806-lvhkg           2/2       Running            0          1h
reviews-v3-1808066742-1pfwk          2/2       Running            0          1h
service-one-2544966823-np65q         2/2       Running            0          1h
service-two-3660184397-r9rvg         2/2       Running            0          1h
servicegraph-168667903-hj510         1/1       Running            0          1h

Jaeger services are running

NAME               CLUSTER-IP   EXTERNAL-IP   PORT(S)                        AGE
details            10.0.0.122   <none>        9080/TCP                       1h
jaeger-agent       None         <none>        5775/UDP,6831/UDP,6832/UDP     19m
jaeger-collector   10.0.0.185   <none>        14267/TCP,14268/TCP,9411/TCP   19m
jaeger-query       10.0.0.94    <pending>     80:32589/TCP                   19m
kubernetes         10.0.0.1     <none>        443/TCP                        1h
productpage        10.0.0.62    <none>        9080/TCP                       1h
ratings            10.0.0.11    <none>        9080/TCP                       1h
reviews            10.0.0.21    <none>        9080/TCP                       1h
service-one        10.0.0.251   <none>        80/TCP                         1h
service-two        10.0.0.145   <none>        80/TCP                         1h
zipkin             None         <none>        9411/TCP                       19m

[ZackButcher]

We should repro w/o any Zipkin components - I wonder if there's something funky in Envoy that makes the swap from Zipkin to Jaegar at runtime break things.

[objectiser]

@redhotpenguin The issue is that the istio.yaml has been changed to use zipkinAddress: zipkin.istio-system:9411 so is expecting the zipkin compatible backend to also be deployed in that namespace. Previously the default namespace was used, so deploying the Jaeger all-in-one template (with its zipkin service) worked.

So just need to specify the namespace when deploying the all-in-one template:

kubectl apply -n istio-system -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml

as mentioned in istio/old_issues_repo#73.

However this has shown up a different problem affecting zipkin instrumentation (whether viewed on zipkin or jaeger UI) - also reported by a user here: istio/old_issues_repo#72

This is caused because the way spans are classified in Envoy changed post 1.4.0, to use local configuration (i.e. OperationName: Egress/Ingress) to determine whether the span was client or server.

@ZackButcher depending on when you wanted to release 0.2, you may want to revert to Envoy 1.4.0. Should I create a separate issue in this repo to outline the issue?

btw - I had a look at the envoy config in the proxy container, and it had no listeners defined. Where does the config for these get added?

[ZackButcher]

@objectiser I don't think we're likely to roll back to Envoy's 1.4 release - we actually just moved forward to a more recent (post-1.4) Envoy SHA a few days ago to pick up some fixes.

The listener config is provided by Pilot: here's the code, here's some testdata examples.

[objectiser]

@ZackButcher Thanks - yeah found it, and submitted istio/old_pilot_repo#1358

[ZackButcher]

Awesome, thanks!

[ldemailly]

expected to be fixed in 0.2.6, please confirm and close ?

[sakshigoel12]

@objectiser @ZackButcher is this issue done?

[ZackButcher]

We have confirmation that Zipkin is working now, I have not had time to look at Jaeger.

[objectiser]

Yes, confirmed 0.2.6 works with Jaeger.

[ZackButcher]

Thanks!