Thanos compactor upload COS bucket error

[wu3396]

Thanos version used:

0.24

Object Storage Provider:

COS

What happened:

upload cos object error, The single object size you upload can not be larger than 5GB

What you expected to happen:

Send correct file size to support multipart uploads

Full logs to relevant components:

level=error ts=2022-02-08T01:42:36.662263384Z caller=compact.go:492 msg="retriable error" err="compaction: group 0@14527284864645982195: upload of 01FVB6MWHT2KA0KXJGT444FYBY failed: upload index: upload file /data/thanos/compact/0@14527284864645982195/01FVB6MWHT2KA0KXJGT444FYBY/index as 01FVB6MWHT2KA0KXJGT444FYBY/index: upload cos object: The single object size you upload can not be larger than 5GB"

[hanjm]

Currently we use simple PUT API https://github.com/thanos-io/thanos/blob/main/pkg/objstore/cos/cos.go#L199, it has 5GB service limit , We need implement using multipart uploads API.

on the other hand, you can use flag --compact.block-max-index-size=5GB to skip compact if block-index-size large than 5GB. https://github.com/thanos-io/thanos/blob/main/cmd/thanos/compact.go#L743

[hanjm]

WIP

[hanjm]

Fixed by #5137, @wu3396 Could you try the latest image?

[wu3396]

The latest image runs without errors
I'm using a flag to skip compression, is it possible to remove the "no-compact-mark.json" in the object store to revalidate?

[hanjm]

@wu3396 Just delete the "no-compact-mark.json" file using coscmd or cloud ui. then the compactor will process the block.

[wu3396]

@hanjm 14GB index file uploaded successfully. Thanks for the help!