Custom Metric Rule Not Triggering Despite Valid Values

[angryxie]

Description
I created a custom metric and confirmed that it is generating values correctly in the UI. However, the alarm rule associated with this metric does not trigger as expected. Below are the details of the configuration and observations:

Steps to Reproduce
metric define: endpoint_latency_2s = from(Endpoint.*).filter(latency > 2000).count();

rule define:

    slow_request_rule:
    expression: sum(endpoint_latency_2s) >= 2
    period: 1
    silence-period: 1
    message: 请求节点1分钟大于2S的请求数达到5-{name}
    hooks:
      - "webhook.local-hook"

ensure configurations are loaded correctly

2025-01-06 06:33:57,012 org.apache.skywalking.oap.server.core.alarm.provider.AlarmRulesWatcher 134 [Apollo-Config-1] INFO  [] - Update alarm rules to Rules(rules=[AlarmRule(alarmRuleName=slow_request_rule, expression=sum(endpoint_latency_2s) >= 2, includeMetrics=[endpoint_latency_2s], includeNames=[], includeNamesRegex=, excludeNames=[], excludeNamesRegex=, period=1, silencePeriod=1, message=请求节点1分钟大于2S的请求数达到5-{name}, tags={}, hooks=[webhook.local-hook], maxTrendRange=0)]

Environment

• OAP Version: 10.1.0
• UI Version: 10.1.0
• Java-agent: 8.16.0

Screenshots

Thank You
Thank you for providing this powerful monitoring tool and for your support. Looking forward to your feedback!

[wu-sheng]

You declared sum two, which means, it needs to happen twice in a time windows.

[angryxie]

I'm confused. According to the documentation:

Expression is calculated based on the metric values (Value1 ~ Value7). For example, expression avg(service_resp_time) > 1000, if the values are 1001, 1001, 1001, 1001, 1001, 1001, 1001, the calculation is ((1001 + 1001 + ... + 1001) / 7) > 1000, and the result would be 1 (true). Then the alarm would be triggered.

in my case,the values like

T1	T2	T3	T4	T5	T6	T7
2	0	0	0	0	0	0

expression: sum(endpoint_latency_2s) >= 2
the calculation is (2+0+0+0+0+0+0)>=2, the result should be 1(true)?

[wu-sheng]

Oh, you are correct. That could work.
You may need to check whether windows has those data.

[wu-sheng]

You set windows as 1, that could fail the check. You should use 2 or 3 at least.
Your example has 7 buckets.

[angryxie]

I set the Period to 5, but there was no change.
The metric values are displayed in the Web UI.

[angryxie]

Okay, I will check again. I am using the debug environment, deployed through SkyWalking-Helm, with only one OAP pod. From what I can see, the OAP logs seem to be fine. I will further check the network connection between the UI and OAP.

[wu-sheng]

Are you using H2 or some extra database? If it's H2, please replace it with a proper database.

[angryxie]

No, I'm using Elasticsearch with version 7.5.2. Could this cause any compatibility issues?

[wu-sheng]

I think elasticsearch is good.

[wu-sheng]

AFAIK,oap -> elasticsearch will have 20-30s delay for data persistent, but that is based on oap is healthy. If it isn't, that could be different issue.
Also, this delay could be much shorter when you have telemetry data keeps coming. In your demo, I guess the traffic doesn't.

[angryxie]

• Setting period > 1 is necessary, and it works fine when set to 2 or above.
• The /healthcheck endpoint returned 404 because the Health Checker module was not activated,Reference
• As you mentioned, the alert page does indeed have a delay, but this delay is acceptable.

Everything seems to be working fine now,Thank you for your response!