Migrate objstore package to use new Go cloud

[cstyan]

The Go team announce the Go cloud packages today: https://blog.golang.org/go-cloud

To reduce the amount of internal code in objstore it could make use of the go-cloud/blob, or if we don't need objstore/inmem then objstore could be replaced entirely.

Curious what you guys think @Bplotka @fabxc, if this would be useful I'd like to work on it.

[bwplotka]

Thanks for willing to help!

I would be very very careful about this. IMO:

• We aim for stability right now, so moving to package that is online only 1d might be not the greatest idea.
• We need to first clearly enumerate advantages and disadvantages of moving to it.

For example:

To reduce the amount of internal code in objstore

...might be not true at the end. There is risk that we will end up writing even more LOC to satisfy our current interface.

To sum up, I think we should definitely wait for this package to be proven in wild (and prod) by users that are willing to experiment more. What do you think? (:

BTW there are lots of way more important issues we could use your help! (: Feel free to look on any issues with prefix label need help

[cstyan]

We aim for stability right now, so moving to package that is online only 1d might be not the greatest idea.

Agreed! They even say it's not 100% stable and shouldn't be used in production yet.

There is risk that we will end up writing even more LOC to satisfy our current interface.

That would be disappointing. I think I'll give it a shot just to see what the current version of library would provide and what the code would look like. No intention of merging though.

BTW there are lots of way more important issues we could use your help! (: Feel free to look on any issues with prefix label need help

Will do :)

[bwplotka]

Awesome, thanks! Closing this for now.

[bwplotka]

I think it's slowly time to reevaluate this (:

[cstyan]

👍 I can look into it, but not for a few days at least

[bwplotka]

Thanks! that would be interesting experiment. CC @domgreen

The main acceptance criteria are:

• Having proposal on how to migrate to blob interface - what changes vs our Bucket interface that would require.

The reasons why we potentially would do it is basically this:

• Adding more providers that would fit into blob interface would allow to have them more useful (and maintained) by anyone who use well known go cloud blob interface. That would ensure Thanos can focus on main goals. They could even migrate into their own repositories.

[domgreen]

I think this is a great idea of getting a proposal document of what we would need to do to transition over.

Overall it will enable us to quickly expand clouds and storage providers based on other implementations of bucket stores and will reduce maintenance on our side, allowing us to as Bartek mentions to focus on Thanos main goals over cross platform storage which is the goal of go cloud.

Updating where the buckets are used should be relatively simple, the harder part will migrating the new providers (Azure, Tencent, Swift) over to the new bucket interface and should be done first to ensure we can support all in flight storage options.

[domgreen]

My initial evaluation of this over this past weekend is that if we can get all new providers to match the interface for blob storage we can then easily refactor the rest of the code.

gocloud even have the refactoring of exists out of the interface that we were looking to do in the future 👍

[bwplotka]

First we need to make sure Thanos is fine with the interface provided by blob. Useful doc: https://github.com/google/go-cloud/blob/master/internal/docs/design.md#design-decisions

The user facing interface for Thanos: https://godoc.org/github.com/google/go-cloud/blob#Bucket
The interface to implement for object storages: https://godoc.org/github.com/google/go-cloud/blob/driver#Bucket

CC @jojohappy (Tencent) @vglafirov (Azure) @sudhi-vm (OpenStack Swift)

[bwplotka]

Bumping this due to #1648 and #678

I think we can start by replacing the implementations of our objstore.Bucket interfaces with go-cloud e.g S3

[bwplotka]

Overall for now still not clear if worth. Mainly due to the extra dependencies it adds, plus we still need to implement our own ones for other storages than Azure, S3, GCS. For dependencies, every SDK has it (e.g Google and AWS), so maybe it's not that bad.

What's really nice is this multi-part support without knowing file size up-front which we miss in 2 providers (S3 via minio-client). This we can implement on our own or just move to AWS SDK.