From 40ed4b5806cb63e30ae6b7e6c3a054bc18d78f20 Mon Sep 17 00:00:00 2001 From: haanhvu Date: Mon, 26 Sep 2022 02:12:00 +0700 Subject: [PATCH 1/4] Receive external labels proposal Signed-off-by: haanhvu --- .../202209-receive-tenant-external-labels.md | 52 +++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 docs/proposals-accepted/202209-receive-tenant-external-labels.md diff --git a/docs/proposals-accepted/202209-receive-tenant-external-labels.md b/docs/proposals-accepted/202209-receive-tenant-external-labels.md new file mode 100644 index 0000000000..cec3bfa2e5 --- /dev/null +++ b/docs/proposals-accepted/202209-receive-tenant-external-labels.md @@ -0,0 +1,52 @@ +--- +type: proposal +title: Allow specifying tenant-specific external labels in Receivers +status: accepted +owner: haanhvu +menu: proposals-accepted +--- + +## 1 Related links/tickets + +* https://github.com/thanos-io/thanos/issues/5434 + +## 2 What problem(s) are we trying to solve? + +We want to add arbitrary external labels to each tenant in Thanos Receiver. + +## 3 How is it going to be helpful? + +This new functionality is helpful in many use cases. Two typical ones are: +* Compact tenant blocks based on tenants’ external labels +* Query data from tenants which share the same external labels + +## 4 Pitfalls of the current solution + +Currently, we can only add external labels to Receivers. Each tenant in a Receiver only gets one external label, which is the tenant ID. This is not flexible enough to do complex compact and query works at the tenant level. + +## 5 What are the possible solution approaches? + +We’re seeing two solution approaches: +* Specify the tenants for each external label in Receiver: Since we already have the functionality of adding external labels to Receiver, is there a way to specify which tenants each external label goes to? Because many tenants may share the same external labels, this may be a straightforward way for users. +* Add external labels to each tenant directly: This way, we treat each tenant independently, which seems to be a more straightforward way in implementation. If goinf this way, we can start with the tenant configuration file. + +## 6 Which solution approach is the most favorable? + +After discussing internally in the issue team (@fpetkovski, @saswatamcode, @haanhvu), we agree the second approach may be the better one to start with since it’s more straightforward in implementation. + +## 7 Potential challenges of the favorable approach? + +We’re seeing two main challenges: +* Support external labels for decoupled routers and ingesters: Receive routers and ingestors are decoupled functionally, but currently they are not decoupled in code. Issue [#5643](https://github.com/thanos-io/thanos/issues/5643) has already been raised for this. With routers and ingestors intermingled in code, we need to find a way to support external labels for RouterOnly and IngestorOnly. One apparent way is to fix [#5643](https://github.com/thanos-io/thanos/issues/5643) at least partially enough before tackling this challenge. +* Handling changes for external labels + +## 8 Implementation plan? + +* RouterIngestor: + - Add a new `external_labels` field in `tenant` field in hasring config + - Map tenant's `external_labels` to tenant's `tenant_id` + - Wire tenant’s `external_labels` to the tenant’s TSDB instance +* RouterOnly +* IngestorOnly + +### 9 Test plan? From 08cc3a2df1d706b244729276e0281b6660ce0eed Mon Sep 17 00:00:00 2001 From: haanhvu Date: Fri, 7 Oct 2022 23:19:09 +0700 Subject: [PATCH 2/4] Restructure and edit proposal's content Signed-off-by: haanhvu --- .../202209-receive-tenant-external-labels.md | 74 ++++++++++++------- 1 file changed, 47 insertions(+), 27 deletions(-) diff --git a/docs/proposals-accepted/202209-receive-tenant-external-labels.md b/docs/proposals-accepted/202209-receive-tenant-external-labels.md index cec3bfa2e5..5346e1e48a 100644 --- a/docs/proposals-accepted/202209-receive-tenant-external-labels.md +++ b/docs/proposals-accepted/202209-receive-tenant-external-labels.md @@ -8,45 +8,65 @@ menu: proposals-accepted ## 1 Related links/tickets -* https://github.com/thanos-io/thanos/issues/5434 +https://github.com/thanos-io/thanos/issues/5434 -## 2 What problem(s) are we trying to solve? +## 2 Why -We want to add arbitrary external labels to each tenant in Thanos Receiver. +We would like to select and query tenant blocks from Receivers in an arbitrary manner. -## 3 How is it going to be helpful? +## 3 Pitfalls of the current solution -This new functionality is helpful in many use cases. Two typical ones are: -* Compact tenant blocks based on tenants’ external labels -* Query data from tenants which share the same external labels +Currently, we can only add external labels to Receiver itself, not to each tenant in the Receiver. There’s currently no way to select and query tenant blocks in an arbitrary manner. -## 4 Pitfalls of the current solution +## 4 Goals -Currently, we can only add external labels to Receivers. Each tenant in a Receiver only gets one external label, which is the tenant ID. This is not flexible enough to do complex compact and query works at the tenant level. +* Allow users to add arbitrary tenants’ external labels in the easiest way possible +* Allow users to change arbitrary tenants’ external labels in the easiest way possible +* Changes in tenants’ external labels are handled correctly +* Backward compatibility (e.g., with Receiver’s external labels) is assured +* Tenants’ external labels are handled separately in RouterIngestor, RouterOnly, and IngestorOnly modes -## 5 What are the possible solution approaches? +## 5 Non-goals -We’re seeing two solution approaches: -* Specify the tenants for each external label in Receiver: Since we already have the functionality of adding external labels to Receiver, is there a way to specify which tenants each external label goes to? Because many tenants may share the same external labels, this may be a straightforward way for users. -* Add external labels to each tenant directly: This way, we treat each tenant independently, which seems to be a more straightforward way in implementation. If goinf this way, we can start with the tenant configuration file. +* Logically split RouterOnly and IngestorOnly modes (Issue [#5643](https://github.com/thanos-io/thanos/issues/5643)): If RouterOnly and IngestorOnly modes are logically split, implementing tenants’ external labels in RouterOnly and IngestorOnly modes would be less challenging. However, the split could be tricky and time-consuming, which might distract us from implementing tenants’ external labels. So we will not do it now. -## 6 Which solution approach is the most favorable? +## 6 Audience -After discussing internally in the issue team (@fpetkovski, @saswatamcode, @haanhvu), we agree the second approach may be the better one to start with since it’s more straightforward in implementation. +Users who need to have cross-tenant information and do cross-tenant activities -## 7 Potential challenges of the favorable approach? +## 7 How -We’re seeing two main challenges: -* Support external labels for decoupled routers and ingesters: Receive routers and ingestors are decoupled functionally, but currently they are not decoupled in code. Issue [#5643](https://github.com/thanos-io/thanos/issues/5643) has already been raised for this. With routers and ingestors intermingled in code, we need to find a way to support external labels for RouterOnly and IngestorOnly. One apparent way is to fix [#5643](https://github.com/thanos-io/thanos/issues/5643) at least partially enough before tackling this challenge. -* Handling changes for external labels +We’ll add a new `external_labels` field in the hasring config. Something like this: + ``` + [ + { + "hashring": "tenant-a", + "endpoints": ["tenant-a-1.metrics.local:19291/api/v1/receive", "tenant-a-2.metrics.local:19291/api/v1/receive"], + "tenants": ["tenant-a"] + "external_labels": ["label-1", "label-2", "label-3"] + }, + ] + ``` +In [MultiTSDB](https://github.com/thanos-io/thanos/blob/4ce3fe19ebb39a308769fb2a9492295b1f113701/pkg/receive/multitsdb.go#L46), each tenant’s external labels will be mapped to its tenant ID. +Each tenant’s external labels will also be extended to the [list of labels](https://github.com/thanos-io/thanos/blob/4ce3fe19ebb39a308769fb2a9492295b1f113701/pkg/store/labelpb/label.go#L282) when the tenant’s TSDB instance is created. +Tenants’ external labels will be first implemented in RouterIngestor, since this is the main mode. +Once tenants’ external labels are implemented in RouterIngestor, we’ll handle changes in tenants’ external labels. So that when users change the values of `external_labels` field in the hasring config, the changes will appear on users’ query. +Finally, we will implement tenants’ external labels in RouterOnly and IngestorOnly modes. +For the tests, the foremost ones are testing backward compatibility (e.g., with Receiver’s external labels), defining one or multiple tenants’ external labels correctly, handling changes in tenants’ external labels correctly, and shipper detecting and uploading tenants’ external labels correctly to block storage. We may add more tests in the future but currently these are the most important ones to do first. -## 8 Implementation plan? +## 8 Implementation plan -* RouterIngestor: - - Add a new `external_labels` field in `tenant` field in hasring config - - Map tenant's `external_labels` to tenant's `tenant_id` - - Wire tenant’s `external_labels` to the tenant’s TSDB instance -* RouterOnly -* IngestorOnly +* Add a new `external_labels` field in the hasring config +* Map tenant ID to tenant’s external labels +* Extend tenant’s external labels to the list of labels when the tenant’s TSDB instance is created +* Implement tenants’ external labels in RouterIngestor +* Handle changes for tenant’s external labels +* Implement tenants’ external labels in RouterOnly +*Implement tenants’ external labels in IngestorOnly -### 9 Test plan? +### 9 Test plan + +* Backward compatibility (e.g., with Receiver’s external labels) +* Define one or multiple tenants’ external labels correctly +* Handle changes in tenants’ external labels correctly +* Shipper detects and uploads tenants’ external labels correctly to block storage From 54e6af0ca1e5b84903818b2db012008d145e664e Mon Sep 17 00:00:00 2001 From: haanhvu Date: Thu, 22 Dec 2022 23:10:15 +0700 Subject: [PATCH 3/4] Update proposal Signed-off-by: haanhvu --- .../202209-receive-tenant-external-labels.md | 81 +++++++++++-------- 1 file changed, 47 insertions(+), 34 deletions(-) diff --git a/docs/proposals-accepted/202209-receive-tenant-external-labels.md b/docs/proposals-accepted/202209-receive-tenant-external-labels.md index 5346e1e48a..0a1818d3d5 100644 --- a/docs/proposals-accepted/202209-receive-tenant-external-labels.md +++ b/docs/proposals-accepted/202209-receive-tenant-external-labels.md @@ -1,6 +1,6 @@ --- type: proposal -title: Allow specifying tenant-specific external labels in Receivers +title: Allow statically specifying tenant-specific external labels in Receivers status: accepted owner: haanhvu menu: proposals-accepted @@ -12,61 +12,74 @@ https://github.com/thanos-io/thanos/issues/5434 ## 2 Why -We would like to select and query tenant blocks from Receivers in an arbitrary manner. +We would like to do cross-tenant activities like grouping tenants' blocks or querying tenants that share the same attributes. Tenant's external labels can help us do those. ## 3 Pitfalls of the current solution -Currently, we can only add external labels to Receiver itself, not to each tenant in the Receiver. There’s currently no way to select and query tenant blocks in an arbitrary manner. +Currently, we can only add external labels to Receiver itself, not to each tenant in the Receiver. So we can't do cross-tenant activities like grouping tenants' blocks or querying tenants that share the same attributes. ## 4 Goals -* Allow users to add arbitrary tenants’ external labels in the easiest way possible -* Allow users to change arbitrary tenants’ external labels in the easiest way possible +* Allow users to statically add arbitrary tenants’ external labels in the easiest way possible +* Allow users to statically change arbitrary tenants’ external labels in the easiest way possible * Changes in tenants’ external labels are handled correctly * Backward compatibility (e.g., with Receiver’s external labels) is assured * Tenants’ external labels are handled separately in RouterIngestor, RouterOnly, and IngestorOnly modes ## 5 Non-goals -* Logically split RouterOnly and IngestorOnly modes (Issue [#5643](https://github.com/thanos-io/thanos/issues/5643)): If RouterOnly and IngestorOnly modes are logically split, implementing tenants’ external labels in RouterOnly and IngestorOnly modes would be less challenging. However, the split could be tricky and time-consuming, which might distract us from implementing tenants’ external labels. So we will not do it now. +* Logically split RouterOnly and IngestorOnly modes (Issue [#5643](https://github.com/thanos-io/thanos/issues/5643)): If RouterOnly and IngestorOnly modes are logically split, implementing tenants’ external labels in RouterOnly and IngestorOnly modes would be less challenging. However, fixing this issue will not be a goal of this proposal, because it's not directly related to tenants' external labels. Regardless, fixing this issue before implementing tenants’ external labels in RouterOnly and IngestorOnly modes would be the best-case scenario. +* Dynamically extract tenants' external labels from time series' data: This proposal only covers statically specifying tenants' external labels. Dynamically receiving and extracting tenants' external labels from time series' data will be added as a follow-up to this proposal. ## 6 Audience -Users who need to have cross-tenant information and do cross-tenant activities +Users who are admin personas and need to perform admin operations on Thanos for multiple tenants ## 7 How -We’ll add a new `external_labels` field in the hasring config. Something like this: - ``` - [ - { - "hashring": "tenant-a", - "endpoints": ["tenant-a-1.metrics.local:19291/api/v1/receive", "tenant-a-2.metrics.local:19291/api/v1/receive"], - "tenants": ["tenant-a"] - "external_labels": ["label-1", "label-2", "label-3"] - }, - ] - ``` -In [MultiTSDB](https://github.com/thanos-io/thanos/blob/4ce3fe19ebb39a308769fb2a9492295b1f113701/pkg/receive/multitsdb.go#L46), each tenant’s external labels will be mapped to its tenant ID. -Each tenant’s external labels will also be extended to the [list of labels](https://github.com/thanos-io/thanos/blob/4ce3fe19ebb39a308769fb2a9492295b1f113701/pkg/store/labelpb/label.go#L282) when the tenant’s TSDB instance is created. -Tenants’ external labels will be first implemented in RouterIngestor, since this is the main mode. -Once tenants’ external labels are implemented in RouterIngestor, we’ll handle changes in tenants’ external labels. So that when users change the values of `external_labels` field in the hasring config, the changes will appear on users’ query. -Finally, we will implement tenants’ external labels in RouterOnly and IngestorOnly modes. -For the tests, the foremost ones are testing backward compatibility (e.g., with Receiver’s external labels), defining one or multiple tenants’ external labels correctly, handling changes in tenants’ external labels correctly, and shipper detecting and uploading tenants’ external labels correctly to block storage. We may add more tests in the future but currently these are the most important ones to do first. +In the hashring config, there will be new field `external_labels`. Something like this: +``` + [ + { + "hashring": "tenant-a-b", + "endpoints": ["127.0.0.1:10901"], + "tenants": ["tenant-a, tenant-b"] + "external_labels": ["key1=value1", "key2=value2", "key3=value3"] + }, + ] +``` + +In Receivers' MultiTSDB, external labels will be extended to each corresponding tenant's label set when the tenant's TSDB is started. + +Next thing we have to do is handling changes for tenants' external labels. That is, whenever users make any changes to tenants' external labels, Receivers' MultiTSDB will update those changes in each corresponding tenant's label set. + +We will handle the cases of hard tenancy first. Once tenants' external labels can be handled in those cases, we will move to soft tenancy cases. + +Tenants’ external labels will be first implemented in RouterIngestor, since this is the most commonly used mode. + +After that, we can implement tenants’ external labels in RouterOnly and IngestorOnly modes. As stated above, the best-case scenario would be logically splitting RouterOnly and IngestorOnl (Issue [#5643](https://github.com/thanos-io/thanos/issues/5643)) before implement tenants’ external labels in each. + +For the tests, the foremost ones are testing defining one or multiple tenants’ external labels correctly, handling changes in tenants’ external labels correctly, backward compatibility with Receiver’s external labels, and shipper detecting and uploading tenants’ external labels correctly to block storage. We may add more tests in the future but currently these are the most important ones to do first. ## 8 Implementation plan -* Add a new `external_labels` field in the hasring config -* Map tenant ID to tenant’s external labels -* Extend tenant’s external labels to the list of labels when the tenant’s TSDB instance is created -* Implement tenants’ external labels in RouterIngestor -* Handle changes for tenant’s external labels +* Add a new `external_labels` field in the hashring config +* Allow MultiTSDB to extend external labels to each corresponding tenant's label set +* Allow MultiTSDB to update each tenant's label set whenever its external labels change +* Handle external labels in soft tenancy cases * Implement tenants’ external labels in RouterOnly -*Implement tenants’ external labels in IngestorOnly +* Implement tenants’ external labels in IngestorOnly ### 9 Test plan -* Backward compatibility (e.g., with Receiver’s external labels) -* Define one or multiple tenants’ external labels correctly -* Handle changes in tenants’ external labels correctly -* Shipper detects and uploads tenants’ external labels correctly to block storage +* Defining one or multiple tenants’ external labels correctly +* Handling changes in tenants’ external labels correctly +* Backward compatibility with Receiver’s external labels +* Shipper detecting and uploading tenants’ external labels correctly to block storage + +### 10 Follow-up + +* Dynamically extract tenants' external labels from time series' data: Once statically specifying tenants' external labels have been implemented and tested successfully and completely, we can think of implementing dynamically receiving and extracting tenants' external labels from time series' data. +* Automatically making use of tenants' external labels: We can think of the most useful use cases with tenants' external labels and whether we should automate any of those use cases. One typical case is automatically grouping new blocks based on tenants' external labels. + +(Both of these are Ben's ideas expressed [here](https://github.com/thanos-io/thanos/pull/5720#pullrequestreview-1167923565).) \ No newline at end of file From f1b19097b6b274849b940866f6f7d05009a89169 Mon Sep 17 00:00:00 2001 From: haanhvu Date: Thu, 22 Dec 2022 23:29:45 +0700 Subject: [PATCH 4/4] Fix doc error Signed-off-by: haanhvu --- .../proposals-accepted/202209-receive-tenant-external-labels.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals-accepted/202209-receive-tenant-external-labels.md b/docs/proposals-accepted/202209-receive-tenant-external-labels.md index 0a1818d3d5..2d9fc2b5a3 100644 --- a/docs/proposals-accepted/202209-receive-tenant-external-labels.md +++ b/docs/proposals-accepted/202209-receive-tenant-external-labels.md @@ -82,4 +82,4 @@ For the tests, the foremost ones are testing defining one or multiple tenants’ * Dynamically extract tenants' external labels from time series' data: Once statically specifying tenants' external labels have been implemented and tested successfully and completely, we can think of implementing dynamically receiving and extracting tenants' external labels from time series' data. * Automatically making use of tenants' external labels: We can think of the most useful use cases with tenants' external labels and whether we should automate any of those use cases. One typical case is automatically grouping new blocks based on tenants' external labels. -(Both of these are Ben's ideas expressed [here](https://github.com/thanos-io/thanos/pull/5720#pullrequestreview-1167923565).) \ No newline at end of file +(Both of these are Ben's ideas expressed [here](https://github.com/thanos-io/thanos/pull/5720#pullrequestreview-1167923565).)