Can Skywalking use metric of Prometheus?

[dtvn-duytq7]

Prometheus takes metric from the app. I want to ask if Skywalking can connect, take the metric and use it from Prometheus?

[wu-sheng]

You could check backend metrics document and satellite document.

1. We could fetch from Prometheus endpoint.
2. Use OpenTelemetry collector to fetch, and use its OpenCensus exporter to forward metrics.

[dtvn-duytq7]

Bạn có thể kiểm tra tài liệu số liệu phụ trợ và tài liệu vệ tinh.

1. Chúng tôi có thể lấy từ điểm cuối Prometheus.
2. Sử dụng trình thu thập OpenTelemetry để tìm nạp và sử dụng trình xuất OpenCensus của nó để chuyển tiếp các chỉ số.

How exactly should I implement this? Correct me if I'm wrong:
I need to install OpenCensus exporter at the Prometheus server to forward all Prom's metrics. Then at Skywalking backend, use OpenTelemetry collector to fetch those metrics, right?
I've just done some searching but only found that OpenCensus exporter do the opposite way, which is collect stats and metrics from other sources then send to Prometheus.

[wu-sheng]

No, the gRPC service has been provided in the document you found, https://skywalking.apache.org/docs/main/v8.7.0/en/setup/backend/metrics-exporter/. The others are your responsibility.

[dtvn-duytq7]

Thanks for the reply, I'm also trying to work the other way around, which is to send SW backend metrics to Prometheus so Prometheus can use those metrics for monitoring, alerting, etc.
As I read from the doc, I found the gRPC metrics exporter function:
http://skywalking.apache.org/docs/main/v8.7.0/en/setup/backend/metrics-exporter/
Can this feature work for what I am trying to achieve?

But I encounter this error:
2021-10-01 14:03:45,430 - org.apache.skywalking.oap.server.exporter.provider.grpc.GRPCExporter - 116 [pool-8-thread-1] ERROR [] - Getting exporter subscription list fails.
io.grpc.StatusRuntimeException: UNIMPLEMENTED: Method not found: MetricExportService/subscription
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.32.1.jar:1.32.1]
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.32.1.jar:1.32.1]
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.32.1.jar:1.32.1]
at org.apache.skywalking.oap.server.exporter.grpc.MetricExportServiceGrpc$MetricExportServiceBlockingStub.subscription(MetricExportServiceGrpc.java:222) ~[exporter-8.7.0.jar:8.7.0]
at org.apache.skywalking.oap.server.exporter.provider.grpc.GRPCExporter.fetchSubscriptionList(GRPCExporter.java:111) ~[exporter-8.7.0.jar:8.7.0]
at org.apache.skywalking.oap.server.exporter.provider.grpc.GRPCExporter.export(GRPCExporter.java:96) ~[exporter-8.7.0.jar:8.7.0]
at org.apache.skywalking.oap.server.core.analysis.worker.ExportWorker.in(ExportWorker.java:46) ~[server-core-8.7.0.jar:8.7.0]
at org.apache.skywalking.oap.server.core.analysis.worker.ExportWorker.in(ExportWorker.java:31) ~[server-core-8.7.0.jar:8.7.0]
at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.lambda$nextWorker$3(MetricsPersistentWorker.java:252) ~[server-core-8.7.0.jar:8.7.0]
at java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_252]
at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.nextWorker(MetricsPersistentWorker.java:251) ~[server-core-8.7.0.jar:8.7.0]
at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.flushDataToStorage(MetricsPersistentWorker.java:232) ~[server-core-8.7.0.jar:8.7.0]
at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.prepareBatch(MetricsPersistentWorker.java:185) ~[server-core-8.7.0.jar:8.7.0]
at org.apache.skywalking.oap.server.core.analysis.worker.PersistenceWorker.buildBatchRequests(PersistenceWorker.java:71) ~[server-core-8.7.0.jar:8.7.0]
at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$null$2(PersistenceTimer.java:117) ~[server-core-8.7.0.jar:8.7.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_252]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_252]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]

What could possibly the reason? I'm still looking for the reason but still no clue.

[wu-sheng]

No, that is for your own grace server. Prometheus is pull mode, exporter is in push mode
Also, from scale perspective, SkyWalking metrics are larger than Prometheus could supports. Don'trecommend you to do, SkyWalking to Prometheus for all metrics