Consider defaulting index.codec in metricbeat ES template to best_compression?

[peterskim12]

I could be wrong on this but I'd imagine that the vast majority of queries on data delivered by Metricbeat would be aggregations rather than for specific documents where _source would be retrieved?

If this is the case, there might not be much of a downside in defaulting the index.codec in the metricbeat.template.json to best_compression. I haven't done any testing to validate the extent of storage reduction but I could run a quick test.

[ruflin]

This is definitively worth to do some testing. Interesting to know is also what would be the impact on indexing speed with best_compression enabled.

[Shaoranlaos]

Can the _source field not be deactivated completly for the metricbeat?
Like it is proposed in the docu of the source field? _source docu
Or has that consequences that i can not see?
Also applies the same to the _all field?

[ruflin]

@Shaoranlaos Definitively discussions worth to have. One problem with disabling _source is that it will not allow to reindex the data in case for example you want to change the structure of the some old fields.

A discussion that is also related to this is the number of default shards: #3431 I think it is important that we keep it configurable as not all users will want the same. Having #3654 in the future will allow to generate the correct template on demand based on the beat config file hopefully.

[Shaoranlaos]

Yes the reindexing is a use case that i overlooked.

How is it with the _all field? Are there similar points i don´t see?

[peterskim12]

@Shaoranlaos You make a good point on the _all field. The _all field is going to be removed in Elasticsearch 6.0 so even without any changes to Beats, this optimization will occur by default in the 6.0 product line. This will result in an improvement in indexing time and index size.

Pre-6.0, disabling the _all field may have downstream impact -- e.g. Kibana's handling of data that doesn't have an _all field.

[tsg]

This was already done for Metricbeat, and with #5095 it's also done for Heartbeat.