[BUG] Error on security events visualization

[AlexRuiz7]

Describe the bug
One of the visualizations of the security events dashboard for the integration with Elastic is failing.

Error message: Query rewrite failed: too many clauses

Query response

{
  "error": {
    "type": "status_exception",
    "reason": "error while executing search",
    "caused_by": {
      "type": "search_phase_execution_exception",
      "reason": "all shards failed",
      "phase": "query",
      "grouped": true,
      "failed_shards": [
        {
          "shard": 0,
          "index": "wazuh-alerts-4.x-2025.02.04",
          "node": "WIqeYvOySr6S-Uj2POy4Aw",
          "reason": {
            "type": "illegal_argument_exception",
            "reason": "Query rewrite failed: too many clauses",
            "caused_by": {
              "type": "too_many_nested_clauses",
              "reason": "Query contains too many nested clauses; maxClauseCount is set to 1024"
            }
          }
        }
      ],
      "caused_by": {
        "type": "illegal_argument_exception",
        "reason": "Query rewrite failed: too many clauses",
        "caused_by": {
          "type": "illegal_argument_exception",
          "reason": "Query rewrite failed: too many clauses",
          "caused_by": {
            "type": "too_many_nested_clauses",
            "reason": "Query contains too many nested clauses; maxClauseCount is set to 1024"
          }
        }
      }
    }
  },
  "rawResponse": {
    "took": 5,
    "timed_out": false,
    "terminated_early": false,
    "num_reduce_phases": 0,
    "_shards": {
      "total": 3,
      "successful": 0,
      "skipped": 0,
      "failed": 3,
      "failures": [
        {
          "shard": 0,
          "index": "wazuh-alerts-4.x-2025.02.04",
          "node": "WIqeYvOySr6S-Uj2POy4Aw",
          "reason": {
            "type": "illegal_argument_exception",
            "reason": "Query rewrite failed: too many clauses",
            "caused_by": {
              "type": "too_many_nested_clauses",
              "reason": "Query contains too many nested clauses; maxClauseCount is set to 1024"
            }
          }
        }
      ]
    },
    "hits": {
      "total": {
        "value": 0,
        "relation": "gte"
      },
      "max_score": null,
      "hits": []
    }
  }
}

Looks like the request has too many nested bool filters

POST /wazuh-alerts-4.x-*/_async_search?batched_reduce_size=64&ccs_minimize_roundtrips=true&wait_for_completion_timeout=200ms&keep_on_completion=true&keep_alive=60000ms&ignore_unavailable=true&preference=1738688902687
{
  "aggs": {
    "0-bucket": {
      "filter": {
        "bool": {
          "must": [],
          "filter": [
            {
              "bool": {
                "should": [
                  {
                    "bool": {
                      "should": [
                        {
                          "term": {
                            "rule.groups": {
                              "value": "authentication_failed"
                            }
                          }
                        }
                      ],
                      "minimum_should_match": 1
                    }
                  },
                  {
                    "multi_match": {
                      "type": "phrase",
                      "query": "win_authentication_failed",
                      "lenient": true
                    }
                  },
                  {
                    "multi_match": {
                      "type": "phrase",
                      "query": "authentication_failures",
                      "lenient": true
                    }
                  }
                ],
                "minimum_should_match": 1
              }
            }
          ],
          "should": [],
          "must_not": []
        }
      },
      "aggs": {
        "0-metric": {
          "cardinality": {
            "field": "rule.groups"
          }
        }
      }
    }
  },
  "size": 0,
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "bool": {
            "must": [],
            "filter": [
              {
                "bool": {
                  "should": [
                    {
                      "bool": {
                        "should": [
                          {
                            "term": {
                              "rule.groups": "authentication_failed"
                            }
                          }
                        ],
                        "minimum_should_match": 1
                      }
                    },
                    {
                      "multi_match": {
                        "type": "phrase",
                        "query": "win_authentication_failed",
                        "lenient": true
                      }
                    },
                    {
                      "multi_match": {
                        "type": "phrase",
                        "query": "authentication_failures",
                        "lenient": true
                      }
                    }
                  ],
                  "minimum_should_match": 1
                }
              }
            ],
            "should": [],
            "must_not": []
          }
        },
        {
          "range": {
            "timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2025-02-04T16:53:31.415Z",
              "lte": "2025-02-04T17:08:31.415Z"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  },
  "stored_fields": [
    "*"
  ],
  "runtime_mappings": {},
  "script_fields": {},
  "fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    },
    {
      "field": "data.aws.createdAt",
      "format": "date_time"
    },
    {
      "field": "data.aws.end",
      "format": "date_time"
    },
    {
      "field": "data.aws.resource.instanceDetails.launchTime",
      "format": "date_time"
    },
    {
      "field": "data.aws.service.eventFirstSeen",
      "format": "date_time"
    },
    {
      "field": "data.aws.service.eventLastSeen",
      "format": "date_time"
    },
    {
      "field": "data.aws.start",
      "format": "date_time"
    },
    {
      "field": "data.aws.updatedAt",
      "format": "date_time"
    },
    {
      "field": "data.timestamp",
      "format": "date_time"
    },
    {
      "field": "data.vulnerability.published",
      "format": "date_time"
    },
    {
      "field": "data.vulnerability.updated",
      "format": "date_time"
    },
    {
      "field": "syscheck.mtime_after",
      "format": "date_time"
    },
    {
      "field": "syscheck.mtime_before",
      "format": "date_time"
    },
    {
      "field": "timestamp",
      "format": "date_time"
    }
  ]
}

To Reproduce
Steps to reproduce the behavior:

1. Go to 'integrations/elastic'
2. Follow the README to create the environment
3. Navigate to the security events dashboard.
4. See error

Expected behavior
No errors on any visualization.

Screenshots

Host/Environment (please complete the following information):

• Integration with Elastic
• Version 8.17.1

Additional context
#675 (comment)

[JuanGarriuz]

I have had various issues getting the environment up and running (mainly related to certificates and Logstash). I had to delete the certificates and copy those from a colleague to avoid certificate-related problems.

I'm not sure if using different workarounds to solve the issue of starting the environment could have somehow affected the Elastic environment test.

[Desvelao]

I could not replicate the problem. I used the 674-integrations-maintenance branch to init the integrations environment.

[AlexRuiz7]

Weird enough, it now works after re-building the environment

[AlexRuiz7]

I'm closing the issue then.

Thank you very much for the troubleshooting @JuanGarriuz @Desvelao