ApplicationDoubleHistogramBuilder cannot be cast to ExtendedDoubleHistogramBuilder

[sebastian-alfers]

Expected behavior

Using the GlobalOpenTelemetry I want to cast a meter.histogramBuilder into a ExtendedDoubleHistogramBuilder to be able to use builder.setAdvice.

Full example:

val meter = GlobalOpenTelemetry.getMeter("OTEL.AzureMonitor.Demo")
val builder = meter
    .histogramBuilder("my-histogram")
    .asInstanceOf[ExtendedDoubleHistogramBuilder]
builder.setAdvice{ advice =>
    advice.setExplicitBucketBoundaries(util.Arrays.asList(10.0, 20.0, 30.0))
  }
val metric = builder.build()

metric.record(i * k)

Actual behavior

This error is thrown:

[info] 2023-06-22 18:50:10.159+02:00 INFO  c.m.applicationinsights.agent - Application Insights Java Agent 3.4.14 started successfully (PID 66902, JVM running for 5.347 s)
...
[error] Exception in thread "main" java.lang.ClassCastException: class io.opentelemetry.javaagent.instrumentation.opentelemetryapi.v1_10.metrics.ApplicationDoubleHistogramBuilder cannot be cast to class io.opentelemetry.extension.incubator.metrics.ExtendedDoubleHistogramBuilder (io.opentelemetry.javaagent.instrumentation.opentelemetryapi.v1_10.metrics.ApplicationDoubleHistogramBuilder and io.opentelemetry.extension.incubator.metrics.ExtendedDoubleHistogramBuilder are in unnamed module of loader 'app')
[error] 	at OpenTelemetry$.delayedEndpoint$OpenTelemetry$1(OpenTelemetry.scala:111)
[error] 	at OpenTelemetry$delayedInit$body.apply(OpenTelemetry.scala:32)

To Reproduce

Run the example above with the latest version of the app insights agent attached.

System information

Please provide the following information:

• SDK Version: applicationinsights-agent-3.4.14.jar
• OS type and version: mac os

Please note that this does not happen if I create OpenTelemetry manually (without the app insights java agent).

The setAdvice api as introduced recently here: open-telemetry/opentelemetry-java#5217.

[heyams]

Hi @sebastian-alfers

Looks like Prototype Histogram advice API was release in 1.26.0.

3.4.10 Java agent is using 1.23.0.
Upgrade to the latest Java agent (at least 3.4.13) should fix your issue.

[sebastian-alfers]

Hi @heyams , thanks for your response. Sorry, this was a typo and I am actually using the latest agent applicationinsights-agent-3.4.14.jar (fixed above now).

[heyams]

@sebastian-alfers no problem. in that case, i will investigate. thanks for the update.

[heyams]

@sebastian-alfers i was able to reproduce your issue.
But i got it working using Azure Monitor OpenTelemetry Exporter. Can you check out this PR ?

all you need is to update CONNECTION_STRING:

private static final String CONNECTION_STRING = "<Your Connection String>";

[heyams]

then you can query customMetrics table on the Application Insights portal:

customMetrics
| where timestamp > ago(5m)
| where name startswith "histogram"

i will add a README for you.

[sebastian-alfers]

Thanks you so much for putting out this effort! It seems to work, and I am getting this two relevant logs:

[2023-06-26 10:19:25,113] [DEBUG] [com.azure.monitor.opentelemetry.exporter.AzureMonitorMetricExporter] [] [PeriodicMetricReader-1] - exporting metric: ImmutableMetricData{resource=Resource{schemaUrl=null, attributes={service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.27.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=my meter, version=null, schemaUrl=null, attributes={}}, name=histogram95d12bac-2646-4403-8af0-355282daa5ed, description=, unit=, type=HISTOGRAM, data=ImmutableHistogramData{aggregationTemporality=DELTA, points=[ImmutableHistogramPointData{getStartEpochNanos=1687767564098648000, getEpochNanos=1687767565107535000, getAttributes={}, getSum=40.0, getCount=3, hasMin=true, getMin=5.0, hasMax=true, getMax=20.0, getBoundaries=[10.0, 20.0, 30.0], getCounts=[1, 2, 0, 0], getExemplars=[]}]}} MDC: {}
[2023-06-26 10:19:25,223] [DEBUG] [com.azure.monitor.opentelemetry.exporter.implementation.pipeline.TelemetryItemExporter] [] [PeriodicMetricReader-1] - sending telemetry to ingestion service:
{"ver":1,"name":"Metric","time":"2023-06-26T08:19:25.107Z","iKey":"34db24f5-d35b-437e-a194-d8ceb8a5960e","tags":{"ai.internal.sdkVersion":"java17.0.4.1:otel1.27.0:ext1.0.0-beta.8","ai.cloud.role":"unknown_service:java"},"data":{"baseType":"MetricData","baseData":{"ver":2,"metrics":[{"name":"histogram95d12bac-2646-4403-8af0-355282daa5ed","value":40.0,"count":3,"min":5.0,"max":20.0}]}}} MDC: {}

The first log line indicates that the custom bucket are taken into account. However I do not see a place in azure where this buckets are being exercised, e.g. when I look at logs with the query you proposed above, or when I build a custom dashboard.

I am happy with the approach you proposed, but it means that the agent should not be used anymore, right? This also means I am loosing the date in "Live metrics" which seem to get pushed once I use the agent itself. Is there any option to also get back the live metrics using the custom approach proposed in the PR linked above?

[sebastian-alfers]

Do you see the chance we get support for custom advice boundaries via the agent at some point? It also seems that the azure-monitor-opentelemetry-exporter is still in beta, is there a timeline on when this will be a final 1.0 release?

[heyams]

Advice boundaries are not supported in Java Agent. Correct, azure-monitor-opentelemetry-exporter is still in beta. We plan to GA in around mid August. If you use the standalone exporter, you don't need to use Java agent anymore. Live metrics doesn't support metrics anyway. azure-monitor-opentelemetry-exporter do not have live metric yet, but we do want to support it before GA. Hope this helps.

[sebastian-alfers]

Thanks!