[BUG] Cannot create monitors/alerts on OpenSearch-Dashboards:2.0.0 #254
Comments
dogukanl
changed the title
|
Hello @dogukanl , Thanks for opening and thanks for onboarding on to 2.0.0 so quickly! This index is hidden by default and I know the security demo makes them system index by default. [ref]. With that said and looking at the stack trace, I wonder if the issue is coming from the OpenSearch Security plugin. I will re-route this to the alerting plugin dashboards repo first and defer to the maintainers insight if it belongs in their repo or the security plugin repo. Thanks again! |
|
Hey @kavilla, Thanks for getting back to me! To give you a little more insight: I have experienced the same issue on my test cluster that I have updated from 1.3.2 to 2.0.0 with no apparent problems by changing container images and some breaking config settings in opensearch.yml. Prior to the update, there was already an .opendistro_security index created by the securityadmin.sh. So neither the securityadmin script nor the demo script has run on that cluster after the update. Thinking this could be caused by the update, I have deployed the fresh 2.0.0 cluster above and still got the same errors. |
|
Hello, I have run into the same issue and currently stuck with same error being thrown no matter whatever tried. I tried multiple notification types and it looks like monitor doesn't like the action block with notification channel set and it doesn't even let you save the config or test fire action with 'send message' and keeps throwing same java.lang.IllegalStateException: Can't get text on a START_OBJECT error. |
|
@AWSHurneyt, @lezzago, seems like a duplicate came up #260. This could be due to the backend plugin but I was trying to see if any breaking changes were called in configs in weren't seeing anything. This wasn't caught in the sanity tests either (at least for the one that I checked prior to something delaying the initial release). Do y'all have some insight here? |
|
@kavilla I think I identified the problem. I updated the issue itself rather than commenting them. You can check out the updates. |
|
Hi @dogukanl, thanks for bringing up this issue. Looking more into this, it seems there is an issue with the frontend plugin. The Once we have a fix in place, we will provide a way to build the artifact here for you to update your OpenSearch 2.0 clusters with to no longer have this issue. |
|
The attached plugin artifact (alertingDashboards-2.0.0.0.zip) can be used to reinstall the alerting plugin. To reinstall the plugin, please follow these steps outlined here. To build the plugin artifact manually, please follow these steps. |
Hi, could you give the instruction to use this in a docker environment? Thank you |
|
I still have the same issue creating cluster level metrics monitor after applying the patch. My Dockerfile: FROM opensearchproject/opensearch-dashboards:2.0.0
RUN bin/opensearch-dashboards-plugin remove alertingDashboards \
&& bin/opensearch-dashboards-plugin install https://github.com/opensearch-project/alerting-dashboards-plugin/files/8835333/alertingDashboards-2.0.0.0.zipMy compose file is the same as above, except I'm using the new image. Edit: I accidentally clicked Close. Can you please re-open it. |
|
@pietrogu, you can reinstall it using: https://opensearch.org/docs/latest/opensearch/install/docker#customize-the-docker-image @dogukanl, did you restart the OpenSearch-Dashboards? Also you may need to refresh your browser cache for the fix to show up |
I'm not positive how the zip was built but we should also open in incognito. The cache-buster mechanism might only be based on the OpenSearch Dashboards build number. |
|
Okay sorry, my bad. It still shows "failed to load destinations" when you click "add trigger" until after you've created your first monitor. So I just assumed it didn't work. When I clicked create the monitor was created successfully and subsequent create actions do not have the error pop up on the bottom right. The issue seems to be solved, except the misleading error message. |
|
@lezzago the issue was accidentaly closed by another user. |
|
Hi everyone, we have now release OpenSearch 2.0.1, which includes the patch to fix this problem. We recommend upgrading the 2.0 clusters to 2.0.1 as there are other fixes mentioned here |
|
i'm using 2.2.1 and i still see this. the cluster was created using the opensearch-k8s-operator and installed in version 1.3.4 , then upgraded to 2.2.1 looking at my browser the reason seems to be this: and the response being |
Cannot create monitors/alerts
With the new update I can create a new notification channel just fine. But when I try to create a monitor, I get the exceptions below. The alerting plugin is looking for the .opendistro-alerting-config index which the cluster does not seem to create automatically like it does with other system indices.
To Reproduce
I am using docker to bring up my cluster:
OpenSearch Version
opensearchproject/opensearch:2.0.0
Dashboards Version
opensearchproject/opensearch-dashboards:2.0.0
Plugins
Host/Environment
Errors
When I click on "Add Trigger" under "Create Monitor"
When I try to create the monitor
UPDATE:
Upon further inspection, I've found that ".opendistro-alerting-config" index not existing before trying to create a monitor does not seem to be the issue, or at least from what I can tell.
I've cloned every repo under opensearch-project and used ag to find where "system_indices" are mentioned and I really couldn't find a function, class or a piece of code that creates all system_indices as a bootstrap process.
Long story short I think the issue is with the API because:
This is okay
It automatically creates the ".opendistro-alerting-config" and the monitor works.
This is NOT OKAY
The "action_execution_policy" map somehow creates a problem and results in the following error:
Similarly
This time "action_execution_scope" is "per_execution" instead of "per_alert". And this version causes:
The last 2 erronous requests are what opensearch dashboards sends (minus the "ui_metadata" key) when you are trying to create a monitor.
I don't know whether this is dashboards not adhering to any API changes or an internal problem where opensearch is meant to parse the "action_execution_policy" maps above. Just my two cents.
UPDATE 2
I think I found the problem. Opensearch excepts a list of strings for "actionable_alerts" key, see. Wheras dashboards send a list of dictionaries such as:
The text was updated successfully, but these errors were encountered: