Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable Thanos Azure client to pass object size when known #2231

Closed
zenador opened this issue Jun 24, 2022 · 3 comments
Closed

Enable Thanos Azure client to pass object size when known #2231

zenador opened this issue Jun 24, 2022 · 3 comments
Labels
enhancement New feature or request storage/azure

Comments

@zenador
Copy link
Contributor

zenador commented Jun 24, 2022

Is your feature request related to a problem? Please describe.

For each API request to the object storage (counted in thanos_objstore_bucket_operations_total metric), the Thanos Azure client makes an additional GetBlobProperties call to retrieve the object size, causing a discrepancy between the number of requests reported by Azure and the previous metric, which is problematic as we bill based on number of requests.

This is also caused by #2232

Describe the solution you'd like

Improve the Thanos Azure client implementation to optionally pass the object size when known so we can avoid an unnecessary GetBlobProperties call. Most of our objects are immutable and this could be cached in-memory as we already do in the chunks cache implementation.

Additional context

For each Get() and GetRange() call to Thanos Azure client, we call GetProperties() to get the size of the object:
https://github.com/thanos-io/thanos/blob/ce84ec521c99493e98fb17ec4bb1358b56a68ebb/pkg/objstore/azure/azure.go#L319-L323

@pracucci
Copy link
Collaborator

This could be address by the work @56quarters is currently testing to reduce Azure client memory utilisation when downloading an object.

@pracucci
Copy link
Collaborator

pracucci commented Jul 6, 2022

An alternative way to do it, is removing the need to have the Thanos azure client knowing the object size, which is what thanos-io/thanos#5451 is doing.

@pracucci
Copy link
Collaborator

No more required, because #2408 has removed the call to fetch object attributes at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request storage/azure
Projects
None yet
Development

No branches or pull requests

2 participants