Update proposal

Signed-off-by: haanhvu <[email protected]>

docs/proposals-accepted/202209-receive-tenant-external-labels.md

@@ -1,6 +1,6 @@
 ---
 type: proposal
-title: Allow specifying tenant-specific external labels in Receivers
+title: Allow statically specifying tenant-specific external labels in Receivers
 status: accepted
 owner: haanhvu
 menu: proposals-accepted
@@ -12,61 +12,74 @@ https://github.com/thanos-io/thanos/issues/5434
 
 ## 2 Why
 
-We would like to select and query tenant blocks from Receivers in an arbitrary manner.
+We would like to do cross-tenant activities like grouping tenants' blocks or querying tenants that share the same attributes. Tenant's external labels can help us do those.
 
 ## 3 Pitfalls of the current solution
 
-Currently, we can only add external labels to Receiver itself, not to each tenant in the Receiver. There’s currently no way to select and query tenant blocks in an arbitrary manner.
+Currently, we can only add external labels to Receiver itself, not to each tenant in the Receiver. So we can't do cross-tenant activities like grouping tenants' blocks or querying tenants that share the same attributes.
 
 ## 4 Goals
 
-* Allow users to add arbitrary tenants’ external labels in the easiest way possible
-* Allow users to change arbitrary tenants’ external labels in the easiest way possible
+* Allow users to statically add arbitrary tenants’ external labels in the easiest way possible
+* Allow users to statically change arbitrary tenants’ external labels in the easiest way possible
 * Changes in tenants’ external labels are handled correctly
 * Backward compatibility (e.g., with Receiver’s external labels) is assured
 * Tenants’ external labels are handled separately in RouterIngestor, RouterOnly, and IngestorOnly modes
 
 ## 5 Non-goals
 
-* Logically split RouterOnly and IngestorOnly modes (Issue [#5643](https://github.com/thanos-io/thanos/issues/5643)): If RouterOnly and IngestorOnly modes are logically split, implementing tenants’ external labels in RouterOnly and IngestorOnly modes would be less challenging. However, the split could be tricky and time-consuming, which might distract us from implementing tenants’ external labels. So we will not do it now.
+* Logically split RouterOnly and IngestorOnly modes (Issue [#5643](https://github.com/thanos-io/thanos/issues/5643)): If RouterOnly and IngestorOnly modes are logically split, implementing tenants’ external labels in RouterOnly and IngestorOnly modes would be less challenging. However, fixing this issue will not be a goal of this proposal, because it's not directly related to tenants' external labels. Regardless, fixing this issue before implementing tenants’ external labels in RouterOnly and IngestorOnly modes would be the best-case scenario.
+* Dynamically extract tenants' external labels from time series' data: This proposal only covers statically specifying tenants' external labels. Dynamically receiving and extracting tenants' external labels from time series' data will be added as a follow-up to this proposal.
 
 ## 6 Audience
 
-Users who need to have cross-tenant information and do cross-tenant activities
+Users who are admin personas and need to perform admin operations on Thanos for multiple tenants
 
 ## 7 How
 
-We’ll add a new `external_labels` field in the hasring config. Something like this:
- ```
- [
-    {
-        "hashring": "tenant-a",
-        "endpoints": ["tenant-a-1.metrics.local:19291/api/v1/receive", "tenant-a-2.metrics.local:19291/api/v1/receive"],
-        "tenants": ["tenant-a"]
-        "external_labels": ["label-1", "label-2", "label-3"]
-    },
- ]
- ```
-In [MultiTSDB](https://github.com/thanos-io/thanos/blob/4ce3fe19ebb39a308769fb2a9492295b1f113701/pkg/receive/multitsdb.go#L46), each tenant’s external labels will be mapped to its tenant ID.
-Each tenant’s external labels will also be extended to the [list of labels](https://github.com/thanos-io/thanos/blob/4ce3fe19ebb39a308769fb2a9492295b1f113701/pkg/store/labelpb/label.go#L282) when the tenant’s TSDB instance is created.
-Tenants’ external labels will be first implemented in RouterIngestor, since this is the main mode.
-Once tenants’ external labels are implemented in RouterIngestor, we’ll handle changes in tenants’ external labels. So that when users change the values of `external_labels` field in the hasring config, the changes will appear on users’ query.
-Finally, we will implement tenants’ external labels in RouterOnly and IngestorOnly modes.
-For the tests, the foremost ones are testing backward compatibility (e.g., with Receiver’s external labels), defining one or multiple tenants’ external labels correctly, handling changes in tenants’ external labels correctly, and shipper detecting and uploading tenants’ external labels correctly to block storage. We may add more tests in the future but currently these are the most important ones to do first.
+In the hashring config, there will be new field `external_labels`. Something like this:
+```
+    [
+        {
+            "hashring": "tenant-a-b",
+            "endpoints": ["127.0.0.1:10901"],
+            "tenants": ["tenant-a, tenant-b"]
+            "external_labels": ["key1=value1", "key2=value2", "key3=value3"]
+        },
+    ]
+```
+
+In Receivers' MultiTSDB, external labels will be extended to each corresponding tenant's label set when the tenant's TSDB is started.
+
+Next thing we have to do is handling changes for tenants' external labels. That is, whenever users make any changes to tenants' external labels, Receivers' MultiTSDB will update those changes in each corresponding tenant's label set.
+
+We will handle the cases of hard tenancy first. Once tenants' external labels can be handled in those cases, we will move to soft tenancy cases.
+
+Tenants’ external labels will be first implemented in RouterIngestor, since this is the most commonly used mode.
+
+After that, we can implement tenants’ external labels in RouterOnly and IngestorOnly modes. As stated above, the best-case scenario would be logically splitting RouterOnly and IngestorOnl (Issue [#5643](https://github.com/thanos-io/thanos/issues/5643)) before implement tenants’ external labels in each.
+
+For the tests, the foremost ones are testing defining one or multiple tenants’ external labels correctly, handling changes in tenants’ external labels correctly, backward compatibility with Receiver’s external labels,  and shipper detecting and uploading tenants’ external labels correctly to block storage. We may add more tests in the future but currently these are the most important ones to do first.
 
 ## 8 Implementation plan
 
-* Add a new `external_labels` field in the hasring config
-* Map tenant ID to tenant’s external labels
-* Extend tenant’s external labels to the list of labels when the tenant’s TSDB instance is created
-* Implement tenants’ external labels in RouterIngestor
-* Handle changes for tenant’s external labels
+* Add a new `external_labels` field in the hashring config
+* Allow MultiTSDB to extend external labels to each corresponding tenant's label set
+* Allow MultiTSDB to update each tenant's label set whenever its external labels change
+* Handle external labels in soft tenancy cases
 * Implement tenants’ external labels in RouterOnly
-*Implement tenants’ external labels in IngestorOnly
+* Implement tenants’ external labels in IngestorOnly
 
 ### 9 Test plan
 
-* Backward compatibility (e.g., with Receiver’s external labels)
-* Define one or multiple tenants’ external labels correctly
-* Handle changes in tenants’ external labels correctly
-* Shipper detects and uploads tenants’ external labels correctly to block storage
+* Defining one or multiple tenants’ external labels correctly
+* Handling changes in tenants’ external labels correctly
+* Backward compatibility with Receiver’s external labels
+* Shipper detecting and uploading tenants’ external labels correctly to block storage
+
+### 10 Follow-up
+
+* Dynamically extract tenants' external labels from time series' data: Once statically specifying tenants' external labels have been implemented and tested successfully and completely, we can think of implementing dynamically receiving and extracting tenants' external labels from time series' data.
+* Automatically making use of tenants' external labels: We can think of the most useful use cases with tenants' external labels and whether we should automate any of those use cases. One typical case is automatically grouping new blocks based on tenants' external labels.
+
+(Both of these are Ben's ideas expressed [here](https://github.com/thanos-io/thanos/pull/5720#pullrequestreview-1167923565).)