Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Jaeger-V2] Add configurable index data layout and rollover frequency #5790

Closed
wants to merge 10 commits into from
Closed

[Jaeger-V2] Add configurable index data layout and rollover frequency #5790

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
47c0a63
Add configurable index data layout and rollover frequency
JaredTan95 Sep 2, 2024
4ce1b97
remove indexprefix
JaredTan95 Sep 3, 2024
037f479
Merge branch 'main' into configure-index-daltalayout-and-frequency
JaredTan95 Sep 3, 2024
2e86cd2
Merge branch 'main' into configure-index-daltalayout-and-frequency
JaredTan95 Sep 6, 2024
cf818b8
fix prefix
JaredTan95 Sep 7, 2024
890daa3
fix prefix
JaredTan95 Sep 8, 2024
3e65f6b
Merge branch 'main' into configure-index-daltalayout-and-frequency
JaredTan95 Sep 8, 2024
f99515e
fix prefix
JaredTan95 Sep 8, 2024
b712a72
Merge branch 'configure-index-daltalayout-and-frequency' of https://g…
JaredTan95 Sep 8, 2024
b9cebd6
fix-opensearch-config
yurishkuro Sep 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 12 additions & 11 deletions cmd/es-rollover/app/init/action.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,18 +29,19 @@ type Action struct {

func (c Action) getMapping(version uint, templateName string) (string, error) {
mappingBuilder := mappings.MappingBuilder{
TemplateBuilder: es.TextTemplateBuilder{},
PrioritySpanTemplate: int64(c.Config.PrioritySpanTemplate),
PriorityServiceTemplate: int64(c.Config.PriorityServiceTemplate),
PriorityDependenciesTemplate: int64(c.Config.PriorityDependenciesTemplate),
PrioritySamplingTemplate: int64(c.Config.PrioritySamplingTemplate),
Shards: int64(c.Config.Shards),
Replicas: int64(c.Config.Replicas),
IndexPrefix: c.Config.IndexPrefix,
UseILM: c.Config.UseILM,
ILMPolicyName: c.Config.ILMPolicyName,
EsVersion: version,
TemplateBuilder: es.TextTemplateBuilder{},
Indices: c.Config.Indices,
UseILM: c.Config.UseILM,
ILMPolicyName: c.Config.ILMPolicyName,
EsVersion: version,
}

// TODO: does not have a separate flag for Indices
mappingBuilder.Indices.Spans.Prefix = c.Config.IndexPrefix
mappingBuilder.Indices.Services.Prefix = c.Config.IndexPrefix
mappingBuilder.Indices.Dependencies.Prefix = c.Config.IndexPrefix
mappingBuilder.Indices.Sampling.Prefix = c.Config.IndexPrefix

return mappingBuilder.GetMapping(templateName)
}

Expand Down
28 changes: 16 additions & 12 deletions cmd/es-rollover/app/init/flags.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
"github.com/spf13/viper"

"github.com/jaegertracing/jaeger/cmd/es-rollover/app"
cfg "github.com/jaegertracing/jaeger/pkg/es/config"
)

const (
Expand All @@ -23,12 +24,7 @@ const (
// Config holds configuration for index cleaner binary.
type Config struct {
app.Config
Shards int
Replicas int
PrioritySpanTemplate int
PriorityServiceTemplate int
PriorityDependenciesTemplate int
PrioritySamplingTemplate int
cfg.Indices
}

// AddFlags adds flags for TLS to the FlagSet.
Expand All @@ -43,10 +39,18 @@ func (*Config) AddFlags(flags *flag.FlagSet) {

// InitFromViper initializes config from viper.Viper.
func (c *Config) InitFromViper(v *viper.Viper) {
c.Shards = v.GetInt(shards)
c.Replicas = v.GetInt(replicas)
c.PrioritySpanTemplate = v.GetInt(prioritySpanTemplate)
c.PriorityServiceTemplate = v.GetInt(priorityServiceTemplate)
c.PriorityDependenciesTemplate = v.GetInt(priorityDependenciesTemplate)
c.PrioritySamplingTemplate = v.GetInt(prioritySamplingTemplate)
c.Indices.Spans.Shards = v.GetInt(shards)
c.Indices.Services.Shards = v.GetInt(shards)
c.Indices.Dependencies.Shards = v.GetInt(shards)
c.Indices.Sampling.Shards = v.GetInt(shards)

c.Indices.Spans.Replicas = v.GetInt(replicas)
c.Indices.Services.Replicas = v.GetInt(replicas)
c.Indices.Dependencies.Replicas = v.GetInt(replicas)
c.Indices.Sampling.Replicas = v.GetInt(replicas)

c.Indices.Spans.Priority = v.GetInt(prioritySpanTemplate)
c.Indices.Services.Priority = v.GetInt(priorityServiceTemplate)
c.Indices.Dependencies.Priority = v.GetInt(priorityDependenciesTemplate)
c.Indices.Sampling.Priority = v.GetInt(prioritySamplingTemplate)
}
12 changes: 6 additions & 6 deletions cmd/es-rollover/app/init/flags_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,10 @@ func TestBindFlags(t *testing.T) {
require.NoError(t, err)

c.InitFromViper(v)
assert.Equal(t, 8, c.Shards)
assert.Equal(t, 16, c.Replicas)
assert.Equal(t, 300, c.PrioritySpanTemplate)
assert.Equal(t, 301, c.PriorityServiceTemplate)
assert.Equal(t, 302, c.PriorityDependenciesTemplate)
assert.Equal(t, 303, c.PrioritySamplingTemplate)
assert.Equal(t, 8, c.Indices.Spans.Shards)
assert.Equal(t, 16, c.Indices.Spans.Replicas)
assert.Equal(t, 300, c.Indices.Spans.Priority)
assert.Equal(t, 301, c.Indices.Services.Priority)
assert.Equal(t, 302, c.Indices.Dependencies.Priority)
assert.Equal(t, 303, c.Indices.Sampling.Priority)
}
8 changes: 4 additions & 4 deletions cmd/esmapping-generator/app/flags.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ import (
type Options struct {
Mapping string
EsVersion uint
Shards int64
Replicas int64
Shards int
Replicas int
IndexPrefix string
UseILM string // using string as util is being used in python and using bool leads to type issues.
ILMPolicyName string
Expand Down Expand Up @@ -40,12 +40,12 @@ func (o *Options) AddFlags(command *cobra.Command) {
esVersionFlag,
7,
"The major Elasticsearch version")
command.Flags().Int64Var(
command.Flags().IntVar(
&o.Shards,
shardsFlag,
5,
"The number of shards per index in Elasticsearch")
command.Flags().Int64Var(
command.Flags().IntVar(
&o.Replicas,
replicasFlag,
1,
Expand Down
9 changes: 5 additions & 4 deletions cmd/esmapping-generator/app/flags_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,9 @@ func TestOptionsWithDefaultFlags(t *testing.T) {

assert.Equal(t, "", o.Mapping)
assert.Equal(t, uint(7), o.EsVersion)
assert.Equal(t, int64(5), o.Shards)
assert.Equal(t, int64(1), o.Replicas)
assert.Equal(t, 5, o.Shards)
assert.Equal(t, 1, o.Replicas)

assert.Equal(t, "", o.IndexPrefix)
assert.Equal(t, "false", o.UseILM)
assert.Equal(t, "jaeger-ilm-policy", o.ILMPolicyName)
Expand All @@ -44,8 +45,8 @@ func TestOptionsWithFlags(t *testing.T) {
require.NoError(t, err)
assert.Equal(t, "jaeger-span", o.Mapping)
assert.Equal(t, uint(7), o.EsVersion)
assert.Equal(t, int64(5), o.Shards)
assert.Equal(t, int64(1), o.Replicas)
assert.Equal(t, 5, o.Shards)
assert.Equal(t, 1, o.Replicas)
assert.Equal(t, "test", o.IndexPrefix)
assert.Equal(t, "true", o.UseILM)
assert.Equal(t, "jaeger-test-policy", o.ILMPolicyName)
Expand Down
21 changes: 15 additions & 6 deletions cmd/esmapping-generator/app/renderer/render.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (

"github.com/jaegertracing/jaeger/cmd/esmapping-generator/app"
"github.com/jaegertracing/jaeger/pkg/es"
cfg "github.com/jaegertracing/jaeger/pkg/es/config"
"github.com/jaegertracing/jaeger/plugin/storage/es/mappings"
)

Expand All @@ -25,14 +26,22 @@ func GetMappingAsString(builder es.TemplateBuilder, opt *app.Options) (string, e
return "", err
}

indexOpts := cfg.IndexOptions{
Prefix: opt.IndexPrefix,
Shards: opt.Shards,
Replicas: opt.Shards,
}
mappingBuilder := mappings.MappingBuilder{
TemplateBuilder: builder,
Shards: opt.Shards,
Replicas: opt.Replicas,
EsVersion: opt.EsVersion,
IndexPrefix: opt.IndexPrefix,
UseILM: enableILM,
ILMPolicyName: opt.ILMPolicyName,
Indices: cfg.Indices{
Spans: indexOpts,
Services: indexOpts,
Dependencies: indexOpts,
Sampling: indexOpts,
},
EsVersion: opt.EsVersion,
UseILM: enableILM,
ILMPolicyName: opt.ILMPolicyName,
}
return mappingBuilder.GetMapping(opt.Mapping)
}
Expand Down
29 changes: 27 additions & 2 deletions cmd/jaeger/config-elasticsearch.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,35 @@ extensions:
backends:
some_storage:
elasticsearch:
index_prefix: "jaeger-main"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: we may want to rethink if we still want a single field in the config for index prefix, which would apply if the individual index options didn't override it explicitly. This would simplify the configuration while maintaining the flexibility.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yurishkuro is this still something we want to address?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be a good usability improvement - perhaps under Indices we can have a "defaults" entry that will apply unless overwritten by the more specific entries like spans:

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yurishkuro this PR is already pretty big (31 files changed). thoughts on getting this PR in and tackling this in a follow-up PR under #6059. There's some follow-up work that needs to be done there anyway for aligning the config with OTEL.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw, it might be a good idea to try out Sapling and do stacked PRs, to avoid precisely this problem (at work we always use stacked PRs)

indices:
JaredTan95 marked this conversation as resolved.
Show resolved Hide resolved
spans:
prefix: "jaeger-main"
date_layout: "2006-01-02"
rollover_frequency: "day"
shards: 5
replicas: 1
services:
prefix: "jaeger-main"
date_layout: "2006-01-02"
rollover_frequency: "day"
shards: 5
replicas: 1
dependencies:
prefix: "jaeger-main"
date_layout: "2006-01-02"
rollover_frequency: "day"
shards: 5
replicas: 1
sampling:
date_layout: "2006-01-02"
rollover_frequency: "day"
shards: 5
replicas: 1
another_storage:
elasticsearch:
index_prefix: "jaeger-archive"
indices:
spans:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: is the services index being built at all when the storage is in the archive mode? I don't think it should be, but in this configuration services.prefix: "jaeger-main" would be the default, and the archive storage is not told in any way that it is indeed archive.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yurishkuro my apologies i don't have a lot of context here but how should we proceed with this?

Copy link
Member

@yurishkuro yurishkuro Oct 6, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to check in the code if writing/reading to/from Archive storage relies exclusively on spans index, or if it uses any other indices. If the latter, then we need to override the prefix to jaeger-archive not just for spans but for other indices too. This is where the defaults: section I mentioned in another comment would be useful - we could just define it once instead of duplicating all the information.

prefix: "jaeger-archive"

receivers:
otlp:
Expand Down
30 changes: 27 additions & 3 deletions cmd/jaeger/config-opensearch.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,35 @@ extensions:
backends:
some_storage:
opensearch:
index_prefix: "jaeger-main"

indices:
spans:
prefix: "jaeger-main"
date_layout: "2006-01-02"
rollover_frequency: "day"
shards: 5
replicas: 1
services:
prefix: "jaeger-main"
date_layout: "2006-01-02"
rollover_frequency: "day"
shards: 5
replicas: 1
dependencies:
prefix: "jaeger-main"
date_layout: "2006-01-02"
rollover_frequency: "day"
shards: 5
replicas: 1
sampling:
date_layout: "2006-01-02"
rollover_frequency: "day"
shards: 5
replicas: 1
another_storage:
opensearch:
index_prefix: "jaeger-main"
indices:
spans:
prefix: "jaeger-archive"

receivers:
otlp:
Expand Down
Loading