copyblocks: support copying between tenants #10110
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andyasp
commented
Dec 3, 2024
tacole02
approved these changes
Dec 3, 2024
Co-authored-by: Taylor C <[email protected]>
narqo
approved these changes
Dec 4, 2024
| @@ -297,7 +329,7 @@ func copyBlocks(ctx context.Context, cfg config, logger log.Logger, m *metrics) | |||
| m.blocksCopied.Inc() | |||
| level.Info(logger).Log("msg", "block copied successfully") | |||
|
|
|||
| err = uploadCopiedMarkerFile(ctx, sourceBucket, tenantID, blockID, destBucket.Name()) | |||
| err = uploadCopiedMarkerFile(ctx, sourceBucket, sourceTenantID, blockID, destBucket.Name()) | |||
There was a problem hiding this comment.
(nit) Maybe we can add a note about the behaviour around markers, just it was clear from the code
Suggested change
| err = uploadCopiedMarkerFile(ctx, sourceBucket, sourceTenantID, blockID, destBucket.Name()) | |
| // Note that markers copying ignores the tenants mapping. This may not work if copying blocks to multiple destination tenants. | |
| err = uploadCopiedMarkerFile(ctx, sourceBucket, sourceTenantID, blockID, destBucket.Name()) |
There was a problem hiding this comment.
I reworded it a bit and added the comment:
// Note that only the blockID and destination bucket are considered in the copy marker.
// If multiple tenants in the same destination bucket are copied to from the same source tenant the markers will currently clash.
tools/copyblocks/main.go
Outdated
| @@ -59,6 +62,7 @@ func (c *config) registerFlags(f *flag.FlagSet) { | |||
| f.DurationVar(&c.copyPeriod, "copy-period", 0, "How often to repeat the copy. If set to 0, copy is done once, and the program stops. Otherwise, the program keeps running and copying blocks until it is terminated.") | |||
| f.Var(&c.enabledUsers, "enabled-users", "If not empty, only blocks for these users are copied.") | |||
| f.Var(&c.disabledUsers, "disabled-users", "If not empty, blocks for these users are not copied.") | |||
| f.Var(&c.crossTenantMapping, "cross-tenant-mapping", "A comma-separated list of (source tenant):(destination tenant). If a source tenant is not mapped then its destination tenant is assumed to be identical.") | |||
There was a problem hiding this comment.
cross-tenant-mapping
(nit) Maybe it's better to stick to "users" on the inputs level? The command has --enabled-users and --disabled-users flags. So it feels confusing if one doesn't know that those are the same. Can we name the flag --user-mapping?
There was a problem hiding this comment.
Nice catch, thanks. I changed it to --user-mapping and updated the documentation to reflect that.
andyasp
added a commit
that referenced
this pull request
Dec 4, 2024
* copyblocks: support copying between tenants * Update tools/copyblocks/README.md * Update tools/copyblocks/README.md Co-authored-by: Taylor C <[email protected]> * Address feedback --------- Co-authored-by: Taylor C <[email protected]> (cherry picked from commit be2f23b) Co-authored-by: Andy Asp <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does
Adds the option
user-mappingtocopyblocksto allow copying between tenants. Previously it was always assumed that blocks would be copied to a different bucket under the same tenant.For example, passing
--user-mapping="foo:bar"would map the blocks from tenantfooto the destination tenantbar.Something I did not touch in this PR was the copy marker naming. It is of the form
markers/blockID-copied-target-bucket. Now that different tenants can be copied to the marker can clash if you copy data from one tenant to multiple tenants in the same destination bucket. That seemed like a weird use case so I deferred to YAGNI.Which issue(s) this PR fixes or relates to
N/A
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX].about-versioning.mdupdated with experimental features.