Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a Ring to IndexGateway #5358

Merged
merged 44 commits into from
Apr 13, 2022
Merged

Add a Ring to IndexGateway #5358

Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
0ba13f0
Begin to add a Ring to IndexGateway
JordanRushing Feb 9, 2022
d928391
Implement missing methods for IndexGateway.
DylanGuedes Feb 10, 2022
325ada2
Fix failing linter
JordanRushing Feb 10, 2022
100e32c
Implement IndexGateway support for dynamic configs.
DylanGuedes Feb 23, 2022
c7797f4
Implement NewBasicService for the IndexGateway.
DylanGuedes Feb 23, 2022
b6d6629
Test IndexGateway dynamic configuration.
DylanGuedes Feb 26, 2022
99df166
Implement new IndexGatewayGRPCPool entity.
DylanGuedes Mar 2, 2022
6eb1595
Make IndexGateway attributes public.
DylanGuedes Mar 2, 2022
0417045
Implement IndexGatewayRing reader.
DylanGuedes Mar 2, 2022
55db3eb
Implement Ring mode in the IndexGatewayClient.
DylanGuedes Mar 7, 2022
d884301
Add new ring index gateway parameter to new Store calls.
DylanGuedes Mar 7, 2022
6cc5034
Use errors.Wrap instead of fmt.Errorf.
DylanGuedes Mar 7, 2022
82393f1
Extract tenantID from context instead of iterating on queries.
DylanGuedes Mar 8, 2022
a4f717c
Remove indexGateway ring param.
DylanGuedes Mar 14, 2022
88224a8
Split IndexGateway server from client implementation.
DylanGuedes Mar 15, 2022
6d68a6c
Fix imports order.
DylanGuedes Mar 15, 2022
947611c
Remove ring as parameter from IndexGateway-related funcs.
DylanGuedes Mar 17, 2022
1004469
Fix default flag value and IndexQuerier type.
DylanGuedes Mar 17, 2022
f0a6b31
Remove additional mode field and reuse it from cfg.
DylanGuedes Mar 17, 2022
cccb298
Remove redundant service init.
DylanGuedes Mar 17, 2022
e824c21
Add sanity check for IndexGateway client constructor.
DylanGuedes Mar 17, 2022
99d39f2
Move mode assigning to initStore method.
DylanGuedes Mar 22, 2022
bd910d9
Reorder IndexGateway constructor.
DylanGuedes Mar 22, 2022
7510b47
Rewrite indexClient chunk.IndexClient as querier Index.Querier.
DylanGuedes Mar 22, 2022
47c4fc2
Fix flag registration for IndexGateway server.
DylanGuedes Mar 22, 2022
3dc51d6
Fix flag registration for test.
DylanGuedes Mar 22, 2022
6323a94
Keep only one reference to indexQuerier.
DylanGuedes Mar 24, 2022
88829bf
Add guard-clause on IndexGatewayRing service.
DylanGuedes Mar 24, 2022
18559e9
Move IndexGatewayClientCfg to gateway_client file.
DylanGuedes Mar 25, 2022
d59f7ee
Merge pull request #13 from JordanRushing/ring-mode-index-gateway
JordanRushing Mar 30, 2022
4629273
Update CHANGELOG.md for `IndexGateway` support for `RingMode`
JordanRushing Mar 30, 2022
5c81c8e
Merge branch 'grafana:main' into add-ring-to-index-gateway
JordanRushing Mar 30, 2022
ebf12c0
Update GatewayClient to use dskit tenant package
JordanRushing Mar 30, 2022
ccbf492
Add listenport configuration for IndexGateway and Ring
JordanRushing Apr 7, 2022
0379649
Make IndexGateway replication factor configurable.
DylanGuedes Apr 8, 2022
a5dda13
Randomize replication set access.
DylanGuedes Apr 8, 2022
bda00af
Merge branch 'main' of github.com:grafana/loki into add-ring-to-index…
DylanGuedes Apr 8, 2022
03c5a7c
Merge branch 'main' of github.com:grafana/loki into add-ring-to-index…
DylanGuedes Apr 11, 2022
62dd62a
Remove unwanted merge HEAD tags.
DylanGuedes Apr 11, 2022
2d602ae
Move away from stores/chunk package.
DylanGuedes Apr 11, 2022
54d6a27
Pass util_log in factory.
DylanGuedes Apr 11, 2022
571b90e
Change index gateway client ring to ignore replicas.
DylanGuedes Apr 11, 2022
6fbd178
Refactor where the common replication factor is applied.
DylanGuedes Apr 11, 2022
b118248
Housekeeping config_wrapper IndexGateway configs.
DylanGuedes Apr 12, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@
* [5393](https://github.com/grafana/loki/pull/5393) **sandeepsukhani**: jsonnet: move boltdb-shipper configs set as compactor args to yaml config
* [5450](https://github.com/grafana/loki/pull/5450) **BenoitKnecht**: pkg/ruler/base: Add external_labels option
* [5484](https://github.com/grafana/loki/pull/5450) **sandeepsukhani**: Add support for per user index query readiness with limits overrides
* [5358](https://github.com/grafana/loki/pull/5358) **DylanGuedes**: Add `RingMode` support to `IndexGateway`

# 2.4.1 (2021/11/07)

Expand Down
48 changes: 38 additions & 10 deletions pkg/loki/config_wrapper.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@ func applyInstanceConfigs(r, defaults *ConfigWrapper) {
r.Ruler.Ring.InstanceAddr = r.Common.InstanceAddr
r.QueryScheduler.SchedulerRing.InstanceAddr = r.Common.InstanceAddr
r.Frontend.FrontendV2.Addr = r.Common.InstanceAddr
r.IndexGateway.Ring.InstanceAddr = r.Common.InstanceAddr
}

if !reflect.DeepEqual(r.Common.InstanceInterfaceNames, defaults.Common.InstanceInterfaceNames) {
Expand All @@ -137,6 +138,7 @@ func applyInstanceConfigs(r, defaults *ConfigWrapper) {
r.Ruler.Ring.InstanceInterfaceNames = r.Common.InstanceInterfaceNames
r.QueryScheduler.SchedulerRing.InstanceInterfaceNames = r.Common.InstanceInterfaceNames
r.Frontend.FrontendV2.InfNames = r.Common.InstanceInterfaceNames
r.IndexGateway.Ring.InstanceInterfaceNames = r.Common.InstanceInterfaceNames
}
}

Expand All @@ -148,7 +150,7 @@ func applyInstanceConfigs(r, defaults *ConfigWrapper) {
// 2. If no explicit ring config is set, use the common ring configured if provided.
// 3. If no common ring was provided, use the memberlist config if provided.
// 4. If no common ring or memberlist were provided, use the ingester's ring configuration.

//
// When using the ingester or common ring config, the loopback interface will be appended to the end of
// the list of default interface names
func applyDynamicRingConfigs(r, defaults *ConfigWrapper) {
Expand All @@ -171,13 +173,13 @@ func applyDynamicRingConfigs(r, defaults *ConfigWrapper) {
}
}

//applyConfigToRings will reuse a given RingConfig everywhere else we have a ring configured.
//`mergeWithExisting` will be true when applying the common config, false when applying the ingester
//config. This decision was made since the ingester ring copying behavior is likely to be less intuitive,
//and was added as a stop-gap to prevent the new rings in 2.4 from breaking existing configs before 2.4 that only had an ingester
//ring defined. When `mergeWithExisting` is false, we will not apply any of the ring config to a ring that has
//any deviations from defaults. When mergeWithExisting is true, the ring config is overlaid on top of any specified
//derivations, with the derivations taking precedence.
// applyConfigToRings will reuse a given RingConfig everywhere else we have a ring configured.
// `mergeWithExisting` will be true when applying the common config, false when applying the ingester
// config. This decision was made since the ingester ring copying behavior is likely to be less intuitive,
// and was added as a stop-gap to prevent the new rings in 2.4 from breaking existing configs before 2.4 that only had an ingester
// ring defined. When `mergeWithExisting` is false, we will not apply any of the ring config to a ring that has
// any deviations from defaults. When mergeWithExisting is true, the ring config is overlaid on top of any specified
// derivations, with the derivations taking precedence.
func applyConfigToRings(r, defaults *ConfigWrapper, rc util.RingConfig, mergeWithExisting bool) {
// Ingester - mergeWithExisting is false when applying the ingester config, and we only want to
// change ingester ring values when applying the common config, so there's no need for the DeepEqual
Expand Down Expand Up @@ -245,6 +247,19 @@ func applyConfigToRings(r, defaults *ConfigWrapper, rc util.RingConfig, mergeWit
r.CompactorConfig.CompactorRing.ZoneAwarenessEnabled = rc.ZoneAwarenessEnabled
r.CompactorConfig.CompactorRing.KVStore = rc.KVStore
}

// IndexGateway
if mergeWithExisting || reflect.DeepEqual(r.StorageConfig.BoltDBShipperConfig.IndexGatewayClientConfig.Ring, defaults.StorageConfig.BoltDBShipperConfig.IndexGatewayClientConfig.Ring) {
r.IndexGateway.Ring.HeartbeatTimeout = rc.HeartbeatTimeout
r.IndexGateway.Ring.HeartbeatPeriod = rc.HeartbeatPeriod
r.IndexGateway.Ring.InstancePort = rc.InstancePort
r.IndexGateway.Ring.InstanceAddr = rc.InstanceAddr
r.IndexGateway.Ring.InstanceID = rc.InstanceID
r.IndexGateway.Ring.InstanceInterfaceNames = rc.InstanceInterfaceNames
r.IndexGateway.Ring.InstanceZone = rc.InstanceZone
r.IndexGateway.Ring.ZoneAwarenessEnabled = rc.ZoneAwarenessEnabled
r.IndexGateway.Ring.KVStore = rc.KVStore
}
}

func applyTokensFilePath(cfg *ConfigWrapper) error {
Expand All @@ -269,6 +284,12 @@ func applyTokensFilePath(cfg *ConfigWrapper) error {
}
cfg.QueryScheduler.SchedulerRing.TokensFilePath = f

f, err = tokensFile(cfg, "indexgateway.tokens")
if err != nil {
return err
}
cfg.IndexGateway.Ring.TokensFilePath = f

return nil
}

Expand Down Expand Up @@ -305,8 +326,10 @@ func applyPathPrefixDefaults(r, defaults *ConfigWrapper) {
}
}

// appendLoopbackInterface will append the loopback interface to the interface names used for the ingester ring,
// v2 frontend, and common ring config unless an explicit list of names was provided.
// appendLoopbackInterface will append the loopback interface to the interface names used by the Loki components
// (ex: rings, v2 frontend, etc).
//
// The append won't occur for an specific component if an explicit list of net interface names is provided for that component.
func appendLoopbackInterface(cfg, defaults *ConfigWrapper) {
loopbackIface, err := loki_net.LoopbackInterfaceName()
if err != nil {
Expand Down Expand Up @@ -340,6 +363,10 @@ func appendLoopbackInterface(cfg, defaults *ConfigWrapper) {
if reflect.DeepEqual(cfg.Ruler.Ring.InstanceInterfaceNames, defaults.Ruler.Ring.InstanceInterfaceNames) {
cfg.Ruler.Ring.InstanceInterfaceNames = append(cfg.Ruler.Ring.InstanceInterfaceNames, loopbackIface)
}

if reflect.DeepEqual(cfg.IndexGateway.Ring.InstanceInterfaceNames, defaults.IndexGateway.Ring.InstanceInterfaceNames) {
cfg.IndexGateway.Ring.InstanceInterfaceNames = append(cfg.IndexGateway.Ring.InstanceInterfaceNames, loopbackIface)
}
}

// applyMemberlistConfig will change the default ingester, distributor, ruler, and query scheduler ring configurations to use memberlist.
Expand All @@ -352,6 +379,7 @@ func applyMemberlistConfig(r *ConfigWrapper) {
r.Ruler.Ring.KVStore.Store = memberlistStr
r.QueryScheduler.SchedulerRing.KVStore.Store = memberlistStr
r.CompactorConfig.CompactorRing.KVStore.Store = memberlistStr
r.IndexGateway.Ring.KVStore.Store = memberlistStr
}

var ErrTooManyStorageConfigs = errors.New("too many storage configs provided in the common config, please only define one storage backend")
Expand Down
35 changes: 34 additions & 1 deletion pkg/loki/config_wrapper_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,9 +182,10 @@ memberlist:
join_members:
- foo.bar.example.com`

config, _ := testContext(configFileString, []string{"-ruler.ring.store", "inmemory"})
config, _ := testContext(configFileString, []string{"-ruler.ring.store", "inmemory", "-index-gateway.ring.store", "etcd"})

assert.EqualValues(t, "inmemory", config.Ruler.Ring.KVStore.Store)
assert.EqualValues(t, "etcd", config.IndexGateway.Ring.KVStore.Store)

assert.EqualValues(t, memberlistStr, config.Ingester.LifecyclerConfig.RingConfig.KVStore.Store)
assert.EqualValues(t, memberlistStr, config.Distributor.DistributorRing.KVStore.Store)
Expand Down Expand Up @@ -948,6 +949,7 @@ common:
assert.Equal(t, "", config.Ingester.LifecyclerConfig.TokensFilePath)
assert.Equal(t, "", config.CompactorConfig.CompactorRing.TokensFilePath)
assert.Equal(t, "", config.QueryScheduler.SchedulerRing.TokensFilePath)
assert.Equal(t, "", config.IndexGateway.Ring.TokensFilePath)
})

t.Run("tokens files should be set from common config when persist_tokens is true and path_prefix is defined", func(t *testing.T) {
Expand All @@ -962,6 +964,7 @@ common:
assert.Equal(t, "/loki/ingester.tokens", config.Ingester.LifecyclerConfig.TokensFilePath)
assert.Equal(t, "/loki/compactor.tokens", config.CompactorConfig.CompactorRing.TokensFilePath)
assert.Equal(t, "/loki/scheduler.tokens", config.QueryScheduler.SchedulerRing.TokensFilePath)
assert.Equal(t, "/loki/indexgateway.tokens", config.IndexGateway.Ring.TokensFilePath)
})

t.Run("ingester config not applied to other rings if actual values set", func(t *testing.T) {
Expand All @@ -975,6 +978,9 @@ compactor:
query_scheduler:
scheduler_ring:
tokens_file_path: /sched/tokes
index_gateway:
ring:
tokens_file_path: /looki/tookens
common:
persist_tokens: true
path_prefix: /loki
Expand All @@ -985,6 +991,7 @@ common:
assert.Equal(t, "/loki/toookens", config.Ingester.LifecyclerConfig.TokensFilePath)
assert.Equal(t, "/foo/tokens", config.CompactorConfig.CompactorRing.TokensFilePath)
assert.Equal(t, "/sched/tokes", config.QueryScheduler.SchedulerRing.TokensFilePath)
assert.Equal(t, "/looki/tookens", config.IndexGateway.Ring.TokensFilePath)
})

t.Run("ingester ring configuration is used for other rings when no common ring or memberlist config is provided", func(t *testing.T) {
Expand All @@ -1003,6 +1010,7 @@ ingester:
assert.Equal(t, "etcd", config.Ruler.Ring.KVStore.Store)
assert.Equal(t, "etcd", config.QueryScheduler.SchedulerRing.KVStore.Store)
assert.Equal(t, "etcd", config.CompactorConfig.CompactorRing.KVStore.Store)
assert.Equal(t, "etcd", config.IndexGateway.Ring.KVStore.Store)
})

t.Run("memberlist configuration takes precedence over copying ingester config", func(t *testing.T) {
Expand All @@ -1025,6 +1033,7 @@ ingester:
assert.Equal(t, "memberlist", config.Ruler.Ring.KVStore.Store)
assert.Equal(t, "memberlist", config.QueryScheduler.SchedulerRing.KVStore.Store)
assert.Equal(t, "memberlist", config.CompactorConfig.CompactorRing.KVStore.Store)
assert.Equal(t, "memberlist", config.IndexGateway.Ring.KVStore.Store)
})
}

Expand Down Expand Up @@ -1175,6 +1184,7 @@ func TestCommonRingConfigSection(t *testing.T) {
assert.Equal(t, "etcd", config.Ruler.Ring.KVStore.Store)
assert.Equal(t, "etcd", config.QueryScheduler.SchedulerRing.KVStore.Store)
assert.Equal(t, "etcd", config.CompactorConfig.CompactorRing.KVStore.Store)
assert.Equal(t, "etcd", config.IndexGateway.Ring.KVStore.Store)
})

t.Run("if common ring is provided, reuse it for all rings that aren't explicitly set", func(t *testing.T) {
Expand All @@ -1196,6 +1206,7 @@ ingester:
assert.Equal(t, "etcd", config.Ruler.Ring.KVStore.Store)
assert.Equal(t, "etcd", config.QueryScheduler.SchedulerRing.KVStore.Store)
assert.Equal(t, "etcd", config.CompactorConfig.CompactorRing.KVStore.Store)
assert.Equal(t, "etcd", config.IndexGateway.Ring.KVStore.Store)
})

t.Run("if only ingester ring is provided, reuse it for all rings", func(t *testing.T) {
Expand All @@ -1211,6 +1222,7 @@ ingester:
assert.Equal(t, "etcd", config.Ruler.Ring.KVStore.Store)
assert.Equal(t, "etcd", config.QueryScheduler.SchedulerRing.KVStore.Store)
assert.Equal(t, "etcd", config.CompactorConfig.CompactorRing.KVStore.Store)
assert.Equal(t, "etcd", config.IndexGateway.Ring.KVStore.Store)
})

t.Run("if a ring is explicitly configured, don't override any part of it with ingester config", func(t *testing.T) {
Expand Down Expand Up @@ -1242,6 +1254,9 @@ ingester:

assert.Equal(t, "inmemory", config.CompactorConfig.CompactorRing.KVStore.Store)
assert.Equal(t, 5*time.Minute, config.CompactorConfig.CompactorRing.HeartbeatPeriod)

assert.Equal(t, "inmemory", config.IndexGateway.Ring.KVStore.Store)
assert.Equal(t, 5*time.Minute, config.IndexGateway.Ring.HeartbeatPeriod)
})

t.Run("if a ring is explicitly configured, merge common config with unconfigured parts of explicitly configured ring", func(t *testing.T) {
Expand Down Expand Up @@ -1273,6 +1288,9 @@ distributor:

assert.Equal(t, "inmemory", config.CompactorConfig.CompactorRing.KVStore.Store)
assert.Equal(t, 5*time.Minute, config.CompactorConfig.CompactorRing.HeartbeatPeriod)

assert.Equal(t, "inmemory", config.IndexGateway.Ring.KVStore.Store)
assert.Equal(t, 5*time.Minute, config.IndexGateway.Ring.HeartbeatPeriod)
})

t.Run("ring configs provided via command line take precedence", func(t *testing.T) {
Expand All @@ -1289,6 +1307,7 @@ distributor:
assert.Equal(t, "consul", config.Ruler.Ring.KVStore.Store)
assert.Equal(t, "consul", config.QueryScheduler.SchedulerRing.KVStore.Store)
assert.Equal(t, "consul", config.CompactorConfig.CompactorRing.KVStore.Store)
assert.Equal(t, "consul", config.IndexGateway.Ring.KVStore.Store)
})

t.Run("common ring config take precedence over common memberlist config", func(t *testing.T) {
Expand All @@ -1306,6 +1325,7 @@ common:
assert.Equal(t, "etcd", config.Ruler.Ring.KVStore.Store)
assert.Equal(t, "etcd", config.QueryScheduler.SchedulerRing.KVStore.Store)
assert.Equal(t, "etcd", config.CompactorConfig.CompactorRing.KVStore.Store)
assert.Equal(t, "etcd", config.IndexGateway.Ring.KVStore.Store)
})
}

Expand Down Expand Up @@ -1369,6 +1389,9 @@ frontend:
compactor:
compactor_ring:
instance_addr: mycompactor
index_gateway:
ring:
instance_addr: myindexgateway
common:
instance_addr: 99.99.99.99
ring:
Expand All @@ -1381,6 +1404,7 @@ common:
assert.Equal(t, "myscheduler", config.QueryScheduler.SchedulerRing.InstanceAddr)
assert.Equal(t, "myqueryfrontend", config.Frontend.FrontendV2.Addr)
assert.Equal(t, "mycompactor", config.CompactorConfig.CompactorRing.InstanceAddr)
assert.Equal(t, "myindexgateway", config.IndexGateway.Ring.InstanceAddr)
})

t.Run("common instance addr is applied when addresses are not explicitly set", func(t *testing.T) {
Expand All @@ -1394,6 +1418,7 @@ common:
assert.Equal(t, "99.99.99.99", config.QueryScheduler.SchedulerRing.InstanceAddr)
assert.Equal(t, "99.99.99.99", config.Frontend.FrontendV2.Addr)
assert.Equal(t, "99.99.99.99", config.CompactorConfig.CompactorRing.InstanceAddr)
assert.Equal(t, "99.99.99.99", config.IndexGateway.Ring.InstanceAddr)
})

t.Run("common instance addr doesn't supersede instance addr from common ring", func(t *testing.T) {
Expand All @@ -1410,6 +1435,7 @@ common:
assert.Equal(t, "22.22.22.22", config.QueryScheduler.SchedulerRing.InstanceAddr)
assert.Equal(t, "99.99.99.99", config.Frontend.FrontendV2.Addr) // not a ring.
assert.Equal(t, "22.22.22.22", config.CompactorConfig.CompactorRing.InstanceAddr)
assert.Equal(t, "22.22.22.22", config.IndexGateway.Ring.InstanceAddr)
})
}

Expand All @@ -1431,6 +1457,10 @@ query_scheduler:
scheduler_ring:
instance_interface_names:
- myscheduler
index_gateway:
ring:
instance_interface_names:
- myindexgateway
frontend:
instance_interface_names:
- myfrontend
Expand All @@ -1452,6 +1482,7 @@ common:
assert.Equal(t, []string{"myscheduler"}, config.QueryScheduler.SchedulerRing.InstanceInterfaceNames)
assert.Equal(t, []string{"myfrontend"}, config.Frontend.FrontendV2.InfNames)
assert.Equal(t, []string{"mycompactor"}, config.CompactorConfig.CompactorRing.InstanceInterfaceNames)
assert.Equal(t, []string{"myindexgateway"}, config.IndexGateway.Ring.InstanceInterfaceNames)
})

t.Run("common instance net interfaces is applied when others net interfaces are not explicitly set", func(t *testing.T) {
Expand All @@ -1466,6 +1497,7 @@ common:
assert.Equal(t, []string{"commoninterface"}, config.QueryScheduler.SchedulerRing.InstanceInterfaceNames)
assert.Equal(t, []string{"commoninterface"}, config.Frontend.FrontendV2.InfNames)
assert.Equal(t, []string{"commoninterface"}, config.CompactorConfig.CompactorRing.InstanceInterfaceNames)
assert.Equal(t, []string{"commoninterface"}, config.IndexGateway.Ring.InstanceInterfaceNames)
})

t.Run("common instance net interface doesn't supersede net interface from common ring", func(t *testing.T) {
Expand All @@ -1484,5 +1516,6 @@ common:
assert.Equal(t, []string{"ringsshouldusethis"}, config.QueryScheduler.SchedulerRing.InstanceInterfaceNames)
assert.Equal(t, []string{"ringsshouldntusethis"}, config.Frontend.FrontendV2.InfNames) // not a ring.
assert.Equal(t, []string{"ringsshouldusethis"}, config.CompactorConfig.CompactorRing.InstanceInterfaceNames)
assert.Equal(t, []string{"ringsshouldusethis"}, config.IndexGateway.Ring.InstanceInterfaceNames)
})
}
10 changes: 8 additions & 2 deletions pkg/loki/loki.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ import (
"github.com/grafana/loki/pkg/storage/chunk"
chunk_storage "github.com/grafana/loki/pkg/storage/chunk/storage"
"github.com/grafana/loki/pkg/storage/stores/shipper/compactor"
"github.com/grafana/loki/pkg/storage/stores/shipper/indexgateway"
"github.com/grafana/loki/pkg/tracing"
"github.com/grafana/loki/pkg/usagestats"
"github.com/grafana/loki/pkg/util"
Expand All @@ -67,6 +68,7 @@ type Config struct {
IngesterClient client.Config `yaml:"ingester_client,omitempty"`
Ingester ingester.Config `yaml:"ingester,omitempty"`
StorageConfig storage.Config `yaml:"storage_config,omitempty"`
IndexGateway indexgateway.Config `yaml:"index_gateway"`
ChunkStoreConfig storage.ChunkStoreConfig `yaml:"chunk_store_config,omitempty"`
SchemaConfig storage.SchemaConfig `yaml:"schema_config,omitempty"`
LimitsConfig validation.Limits `yaml:"limits_config,omitempty"`
Expand Down Expand Up @@ -104,6 +106,7 @@ func (c *Config) RegisterFlags(f *flag.FlagSet) {
c.IngesterClient.RegisterFlags(f)
c.Ingester.RegisterFlags(f)
c.StorageConfig.RegisterFlags(f)
c.IndexGateway.RegisterFlags(f)
c.ChunkStoreConfig.RegisterFlags(f)
c.SchemaConfig.RegisterFlags(f)
c.LimitsConfig.RegisterFlags(f)
Expand Down Expand Up @@ -250,6 +253,7 @@ type Loki struct {
QueryFrontEndTripperware basetripper.Tripperware
queryScheduler *scheduler.Scheduler
usageReport *usagestats.Reporter
indexGatewayRing *ring.Ring

clientMetrics chunk_storage.ClientMetrics

Expand Down Expand Up @@ -486,6 +490,7 @@ func (t *Loki) setupModuleManager() error {
mm.RegisterModule(Compactor, t.initCompactor)
mm.RegisterModule(IndexGateway, t.initIndexGateway)
mm.RegisterModule(QueryScheduler, t.initQueryScheduler)
mm.RegisterModule(IndexGatewayRing, t.initIndexGatewayRing, modules.UserInvisibleModule)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the reason this needs to be a top level module (vs. having the logic to enable/disable the ring be inside the initIndexGateway code) because it's a shared dependency between both the Store and the IndexGateway? Can those module be enabled/disabled independent of one another?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For context: there's an index gateway server and index gateway clients. The clients needs a read ring (provided by the IndexGatewayRing service), while the server needs the whole index gateway service. So, the server needs way more stuff than the other services (ex: the store) as they only need a ring, and sharing the whole server with the store was too much.

Now, to your question: yes, this had to be a top-level module, but not because the ring is shared between the gateway server and client, because although both use the ring, they use it differently (ex: the client only requires a read ring, the server requires a lifecycler+a normal ring). This had to be a top-level module to isolate a very lean read ring, used by the clients.

To your last question: Yes, they can be disabled/independent because you can run an index gateway ring without any clients connecting to it, and you can also have an index gateway client without any server registered in the ring and it will still run - but won't communicate with any index gateway.

mm.RegisterModule(UsageReport, t.initUsageReport)

mm.RegisterModule(All, nil)
Expand All @@ -500,7 +505,7 @@ func (t *Loki) setupModuleManager() error {
OverridesExporter: {Overrides, Server},
TenantConfigs: {RuntimeConfig},
Distributor: {Ring, Server, Overrides, TenantConfigs, UsageReport},
Store: {Overrides},
Store: {Overrides, IndexGatewayRing},
Ingester: {Store, Server, MemberlistKV, TenantConfigs, UsageReport},
Querier: {Store, Ring, Server, IngesterQuerier, TenantConfigs, UsageReport},
QueryFrontendTripperware: {Server, Overrides, TenantConfigs},
Expand All @@ -509,8 +514,9 @@ func (t *Loki) setupModuleManager() error {
Ruler: {Ring, Server, Store, RulerStorage, IngesterQuerier, Overrides, TenantConfigs, UsageReport},
TableManager: {Server, UsageReport},
Compactor: {Server, Overrides, MemberlistKV, UsageReport},
IndexGateway: {Server, Overrides, UsageReport},
IndexGateway: {Server, Overrides, UsageReport, MemberlistKV},
IngesterQuerier: {Ring},
IndexGatewayRing: {RuntimeConfig, Server, MemberlistKV},
All: {QueryScheduler, QueryFrontend, Querier, Ingester, Distributor, Ruler, Compactor},
Read: {QueryScheduler, QueryFrontend, Querier, Ruler, Compactor},
Write: {Ingester, Distributor},
Expand Down
Loading