Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

A Dynamic template with two path_match fails in TSDB #104400

Closed
gsantoro opened this issue Jan 16, 2024 · 1 comment · Fixed by #104418
Closed

A Dynamic template with two path_match fails in TSDB #104400

gsantoro opened this issue Jan 16, 2024 · 1 comment · Fixed by #104418
Labels
>bug :StorageEngine/TSDB You know, for Metrics Team:StorageEngine

Comments

@gsantoro
Copy link

gsantoro commented Jan 16, 2024 •
edited
Loading

Elasticsearch Version

8.11.3

Installed Plugins

No response

Java Version

bundled

OS Version

Linux e3f4b5161783 6.5.11-linuxkit #1 SMP PREEMPT Wed Dec 6 17:08:31 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

Problem Description

This problem only arises with index templates where index.mode: time_series, and a dynamic template has more than one path_match. Time series is only enabled for Metrics types. The same issue doesn't happen for logs data streams.

The same problem doesn't happen if instead of TSDB I only use "_source": { "mode": "synthetic" }.

This problem prevents me from adding ecs@mappings as a component template to all index templates.

More info about what I am trying to achieve at elastic/kibana#174905

The same problem described in this issue appears when testing with the latest ES snapshot 8.13.0-yfnfxbo3-SNAPSHOT

Steps to Reproduce

Try to create a sample index template with this command from the dev console.

put /_index_template/metrics-nginx.stubstatus_test
{
  "index_patterns": [
    "metrics-nginx.stubstatus_test-*"
  ],
  "template": {
    "settings": {
      "index": {
        "mode": "time_series"
      }
    },
    "mappings": {
      "properties": {
        "tags": {
          "type": "keyword",
          "time_series_dimension": true
        }
      },
      "dynamic_templates": [
        {
          "ecs_path_match_keyword_and_match_only_text": {
            "path_match": [
              "test",
              "test2"
            ],
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "keyword"
            }
          }
        }
      ]
    }
  },
  "priority": 200,
  "data_stream": {
    "hidden": false,
    "allow_custom_routing": false
  }
}

You will get this error

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "no type specified for property [text]"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "no type specified for property [text]"
  },
  "status": 400
}

even if the type is specified correctly

The same problem doesn't happen if I only have one path_match like the following

put /_index_template/metrics-nginx.stubstatus_test
{
  "index_patterns": [
    "metrics-nginx.stubstatus_test-*"
  ],
  "template": {
    "settings": {
      "index": {
        "mode": "time_series"
      }
    },
    "mappings": {
      "properties": {
        "tags": {
          "type": "keyword",
          "time_series_dimension": true
        }
      },
      "dynamic_templates": [
        {
          "ecs_path_match_keyword_and_match_only_text": {
            "path_match": [
              "test2"
            ],
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "keyword"
            }
          }
        }
      ]
    }
  },
  "priority": 200,
  "data_stream": {
    "hidden": false,
    "allow_custom_routing": false
  }
}

the stack trace for this error is

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "no type specified for property [text]",
        "stack_trace": """org.elasticsearch.index.mapper.MapperParsingException: no type specified for property [text]
	at [email protected]/org.elasticsearch.index.mapper.TypeParsers.parseMultiField(TypeParsers.java:158)
	at [email protected]/org.elasticsearch.index.mapper.FieldMapper$Builder.parse(FieldMapper.java:1303)
	at [email protected]/org.elasticsearch.index.mapper.FieldMapper$TypeParser.parse(FieldMapper.java:1465)
	at [email protected]/org.elasticsearch.index.mapper.FieldMapper$TypeParser.parse(FieldMapper.java:1422)
	at [email protected]/org.elasticsearch.datastreams.DataStreamIndexSettingsProvider.findRoutingPaths(DataStreamIndexSettingsProvider.java:185)
	at [email protected]/org.elasticsearch.datastreams.DataStreamIndexSettingsProvider.getAdditionalIndexSettings(DataStreamIndexSettingsProvider.java:118)
	at [email protected]/org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.validateIndexTemplateV2(MetadataIndexTemplateService.java:691)
	at [email protected]/org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.addIndexTemplateV2(MetadataIndexTemplateService.java:625)
	at [email protected]/org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.addIndexTemplateV2(MetadataIndexTemplateService.java:562)
	at [email protected]/org.elasticsearch.cluster.metadata.MetadataIndexTemplateService$5.execute(MetadataIndexTemplateService.java:507)
	at [email protected]/org.elasticsearch.cluster.metadata.MetadataIndexTemplateService$1.executeTask(MetadataIndexTemplateService.java:142)
	at [email protected]/org.elasticsearch.cluster.metadata.MetadataIndexTemplateService$1.executeTask(MetadataIndexTemplateService.java:139)
	at [email protected]/org.elasticsearch.cluster.SimpleBatchedExecutor.execute(SimpleBatchedExecutor.java:70)
	at [email protected]/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1039)
	at [email protected]/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1004)
	at [email protected]/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:232)
	at [email protected]/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1626)
	at [email protected]/org.elasticsearch.action.ActionListener.run(ActionListener.java:368)
	at [email protected]/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1623)
	at [email protected]/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)
	at [email protected]/org.elasticsearch.action.ActionListener.run(ActionListener.java:368)
	at [email protected]/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)
	at [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
	at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
"""
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "no type specified for property [text]",
    "stack_trace": """org.elasticsearch.index.mapper.MapperParsingException: no type specified for property [text]
	at [email protected]/org.elasticsearch.index.mapper.TypeParsers.parseMultiField(TypeParsers.java:158)
	at [email protected]/org.elasticsearch.index.mapper.FieldMapper$Builder.parse(FieldMapper.java:1303)
	at [email protected]/org.elasticsearch.index.mapper.FieldMapper$TypeParser.parse(FieldMapper.java:1465)
	at [email protected]/org.elasticsearch.index.mapper.FieldMapper$TypeParser.parse(FieldMapper.java:1422)
	at [email protected]/org.elasticsearch.datastreams.DataStreamIndexSettingsProvider.findRoutingPaths(DataStreamIndexSettingsProvider.java:185)
	at [email protected]/org.elasticsearch.datastreams.DataStreamIndexSettingsProvider.getAdditionalIndexSettings(DataStreamIndexSettingsProvider.java:118)
	at [email protected]/org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.validateIndexTemplateV2(MetadataIndexTemplateService.java:691)
	at [email protected]/org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.addIndexTemplateV2(MetadataIndexTemplateService.java:625)
	at [email protected]/org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.addIndexTemplateV2(MetadataIndexTemplateService.java:562)
	at [email protected]/org.elasticsearch.cluster.metadata.MetadataIndexTemplateService$5.execute(MetadataIndexTemplateService.java:507)
	at [email protected]/org.elasticsearch.cluster.metadata.MetadataIndexTemplateService$1.executeTask(MetadataIndexTemplateService.java:142)
	at [email protected]/org.elasticsearch.cluster.metadata.MetadataIndexTemplateService$1.executeTask(MetadataIndexTemplateService.java:139)
	at [email protected]/org.elasticsearch.cluster.SimpleBatchedExecutor.execute(SimpleBatchedExecutor.java:70)
	at [email protected]/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1039)
	at [email protected]/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1004)
	at [email protected]/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:232)
	at [email protected]/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1626)
	at [email protected]/org.elasticsearch.action.ActionListener.run(ActionListener.java:368)
	at [email protected]/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1623)
	at [email protected]/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)
	at [email protected]/org.elasticsearch.action.ActionListener.run(ActionListener.java:368)
	at [email protected]/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)
	at [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
	at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
"""
  },
  "status": 400
}

Logs (if relevant)
@gsantoro gsantoro added >bug needs:triage Requires assignment of a team area label labels Jan 16, 2024
@felixbarny felixbarny added the :StorageEngine/TSDB You know, for Metrics label Jan 16, 2024
@pxsalehi pxsalehi added Team:StorageEngine and removed needs:triage Requires assignment of a team area label labels Jan 16, 2024
@felixbarny felixbarny mentioned this issue Jan 16, 2024
Merged
@gsantoro
Copy link
Author

Reference meta issue elastic/integrations#8542

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
>bug :StorageEngine/TSDB You know, for Metrics Team:StorageEngine
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix routing_path when template has multiple path_match and multi-fields felixbarny/elasticsearch
3 participants
@felixbarny @gsantoro @pxsalehi