bucket verify ignoring deletion-mark

[AndrewNiven1]

Thanos, Prometheus and Golang version used:

Thanos v0.25.0

Object Storage Provider:

Azure

What happened:

When running thanos tools bucket verify... against an object storage that is actively compacted, it returns a large number of "warning" logs containing information on overlapped blocks. On closer inspection, these overlaps are between blocks of different levels of compaction but contain the same data - for example, a Level 3 block overlapping with the Level 2 blocks that it was made out of. As long as the lower level blocks are marked for deletion (and are not older than expected), this is normal behaviour.

What you expected to happen:

thanos tools bucket verify would ignore blocks that are marked for deletion for the purpose of checking overlapping blocks.

How to reproduce it (as minimally and precisely as possible):

Run thanos tools bucket verify against an object storage that has an active compactor running against it.

Full logs to relevant components:

The full output of thanos tools bucket verify is too long to post, but an excerpt is included below.

level=info ts=2022-03-10T11:59:23.916468249Z caller=factory.go:49 msg="loading bucket configuration"
level=info ts=2022-03-10T11:59:24.721515048Z caller=verify.go:138 verifiers=overlapped_blocks,index_known_issues msg="Starting verify task"
level=info ts=2022-03-10T11:59:24.72215079Z caller=overlapped_blocks.go:29 verifiers=overlapped_blocks,index_known_issues verifier=overlapped_blocks msg="started verifying issue"
level=info ts=2022-03-10T11:59:50.461447174Z caller=fetcher.go:470 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=25.739277031s duration_ms=25739 cached=2207 returned=2207 partial=0
level=warn ts=2022-03-10T11:59:50.48308087Z caller=overlapped_blocks.go:42 verifiers=overlapped_blocks,index_known_issues verifier=overlapped_blocks msg="found overlapped blocks" group=0@6525695680182316826 overlap="[mint: 1646791200006, maxt: 1646798400000, range: 1h59m59s, blocks: 2]: <ulid: 01FXQEP1YAB9GP2AXN5BEGXR1Q, mint: 1646784000020, maxt: 1646812800000, range: 7h59m59s>, <ulid: 01FXPGRP93WAMMG04X9TFKBNSB, mint: 1646791200006, maxt: 1646798400000, range: 1h59m59s>\n[mint: 1646805600025, maxt: 1646812800000, range: 1h59m59s, blocks: 2]: <ulid: 01FXQEP1YAB9GP2AXN5BEGXR1Q, mint: 1646784000020, maxt: 1646812800000, range: 7h59m59s>, <ulid: 01FXPYG4S6XYXJ7VH02JPT5KEY, mint: 1646805600025, maxt: 1646812800000, range: 1h59m59s>\n[mint: 1646848800039, maxt: 1646856000000, range: 1h59m59s, blocks: 2]: <ulid: 01FXS5B2W113AR7SBNN89AB3VS, mint: 1646841600002, maxt: 1646870400000, range: 7h59m59s>, <ulid: 01FXR7PG94KY3MRTVK3YHG7P64, mint: 1646848800039, maxt: 1646856000000, range: 1h59m59s>\n[mint: 1646697600005, maxt: 1646704800000, range: 1h59m59s, blocks: 3]: <ulid: 01FXPKR2PK2SABN058FTYN5AD1, mint: 1646611200003, maxt: 1646784000000, range: 47h59m59s>, <ulid: 01FXMW80288Y81DCNFPMYBCS2J, mint: 1646697600005, maxt: 1646726400000, range: 7h59m59s>, <ulid: 01FXKQG814VJW3G00MR9C5S8Z4, mint: 1646697600005, maxt: 1646704800000, range: 1h59m59s>\n[mint: 1646704800008, maxt: 1646712000000, range: 1h59m59s, blocks: 3]: <ulid: 01FXPKR2PK2SABN058FTYN5AD1, mint: 1646611200003, maxt: 1646784000000, range: 47h59m59s>, <ulid: 01FXMW80288Y81DCNFPMYBCS2J, mint: 1646697600005, maxt: 1646726400000, range: 7h59m59s>, <ulid: 01FXKYBZ94CCRJG0FDJ3NP5JVK, mint: 1646704800008, maxt: 1646712000000, range: 1h59m59s>\n[mint: 1646856000011, maxt: 1646863200000, range: 1h59m59s, blocks: 2]: <ulid: 01FXS5B2W113AR7SBNN89AB3VS, mint: 1646841600002, maxt: 1646870400000, range: 7h59m59s>, <ulid: 01FXREJ7NR14F150G26NHJKTF7, mint: 1646856000011, maxt: 1646863200000, range: 1h59m59s>\n[mint: 1646712000006, maxt: 1646719200000, range: 1h59m59s, blocks: 3]: <ulid: 01FXPKR2PK2SABN058FTYN5AD1, mint: 1646611200003, maxt: 1646784000000, range: 47h59m59s>, <ulid: 01FXMW80288Y81DCNFPMYBCS2J, mint: 1646697600005, maxt: 1646726400000, range: 7h59m59s>, <ulid: 01FXM57PH4R3JVFY4APWB6CRZ7, mint: 1646712000006, maxt: 1646719200000, range: 1h59m59s>\n[mint: 1646740800002, maxt: 1646748000000, range: 1h59m59s, blocks: 3]: <ulid: 01FXPKR2PK2SABN058FTYN5AD1, mint: 1646611200003, maxt: 1646784000000, range: 47h59m59s>, <ulid: 01FXNQ98PXRBDX8E18ZE5Q0E62, mint: 1646726400010, maxt: 1646755200000, range: 7h59m59s>, <ulid: 01FXN0PKH4G1VXRRRCV0RTQJ89, mint: 1646740800002, maxt: 1646748000000, range: 1h59m59s>\n[mint: 1646784000020, maxt: 1646791200000, range: 1h59m59s, blocks: 2]: <ulid: 01FXP9WZ12A7712KXZRMNFJTFD, mint: 1646784000020, maxt: 1646791200000, range: 1h59m59s>, <ulid: 01FXQEP1YAB9GP2AXN5BEGXR1Q, mint: 1646784000020, maxt: 1646812800000, range: 7h59m59s>\n[mint: 1646834400002, maxt: 1646841600000, range: 1h59m59s, blocks: 2]: <ulid: 01FXR9QTH8ZHMAEH0GK5FGEJ1F, mint: 1646812800009, maxt: 1646841600000, range: 7h59m59s>, <ulid: 01FXQSZ1S48KZPP2T05CHGEWRT, mint: 1646834400002, maxt: 1646841600000, range: 1h59m59s>\n[mint: 1646863200010, maxt: 1646870400000, range: 1h59m59s, blocks: 2]: <ulid: 01FXS5B2W113AR7SBNN89AB3VS, mint: 1646841600002, maxt: 1646870400000, range: 7h59m59s>, <ulid: 01FXRNDYS3NA96JJTCJ3GPAMC5, mint: 1646863200010, maxt: 1646870400000, range: 1h59m59s>\n[mint: 1646748000016, maxt: 1646755200000, range: 1h59m59s, blocks: 3]: <ulid: 01FXPKR2PK2SABN058FTYN5AD1, mint: 1646611200003, maxt: 1646784000000, range: 47h59m59s>, <ulid: 01FXNQ98PXRBDX8E18ZE5Q0E62, mint: 1646726400010, maxt: 1646755200000, range: 7h59m59s>, <ulid: 01FXN7JAS2GVWYEKSF9MP45K9K, mint: 1646748000016, maxt: 1646755200000, range: 1h59m59s>\n[mint: 1646719200027, maxt: 1646726400000, range: 1h59m59s, blocks: 3]: <ulid: 01FXPKR2PK2SABN058FTYN5AD1, mint: 1646611200003, maxt: 1646784000000, range: 47h59m59s>, <ulid: 01FXMW80288Y81DCNFPMYBCS2J, mint: 1646697600005, maxt: 1646726400000, range: 7h59m59s>, <ulid: 01FXMC3DS51RJ1H8FX366R561A, mint: 1646719200027, maxt: 1646726400000, range: 1h59m59s>\n[mint: 1646769600001, maxt: 1646776800000, range: 1h59m59s, blocks: 3]: <ulid: 01FXPKR2PK2SABN058FTYN5AD1, mint: 1646611200003, maxt: 1646784000000, range: 47h59m59s>, <ulid: 01FXPJW1QC5J2ER4D9GCC55S1K, mint: 1646755200010, maxt: 1646784000000, range: 7h59m59s>, <ulid: 01FXNW5GH2A6MCQSJ6J6XNH21D, mint: 1646769600001, maxt: 1646776800000, range: 1h59m59s>\n[mint: 1646776800017, maxt: 1646784000000, range: 1h59m59s, blocks: 3]: <ulid: 01FXPKR2PK2SABN058FTYN5AD1, mint: 1646611200003, maxt: 1646784000000, range: 47h59m59s>, <ulid: 01FXPJW1QC5J2ER4D9GCC55S1K, mint: 1646755200010, maxt: 1646784000000, range: 7h59m59s>, <ulid: 01FXP317S55917B6QKZJZ031KN, mint: 1646776800017, maxt: 1646784000000, range: 1h59m59s>\n[mint: 1646611200003, maxt: 1646640000000, range: 7h59m59s, blocks: 2]: <ulid: 01FXPKR2PK2SABN058FTYN5AD1, mint: 1646611200003, maxt: 1646784000000, range: 47h59m59s>, <ulid: 01FXJ9VR7EF9Y0WEWQ04PKP6MN, mint: 1646611200003, maxt: 1646640000000, range: 7h59m59s>\n[mint: 1646726400010, maxt: 1646733600000, range: 1h59m59s, blocks: 3]: <ulid: 01FXPKR2PK2SABN058FTYN5AD1, mint: 1646611200003, maxt: 1646784000000, range: 47h59m59s>, <ulid: 01FXNQ98PXRBDX8E18ZE5Q0E62, mint: 1646726400010, maxt: 1646755200000, range: 7h59m59s>, <ulid: 01FXMJZ514CJSRZ6AQEK1NYVXK, mint: 1646726400010, maxt: 1646733600000, range: 1h59m59s>\n[mint: 1646841600002, maxt: 1646848800000, range: 1h59m59s, blocks: 2]: <ulid: 01FXS5B2W113AR7SBNN89AB3VS, mint: 1646841600002, maxt: 1646870400000, range: 7h59m59s>, <ulid: 01FXR0TS1AX2FV8MKVY8WEK9KB, mint: 1646841600002, maxt: 1646848800000, range: 1h59m59s>\n[mint: 1646640000008, maxt: 1646668800000, range: 7h59m59s, blocks: 2]: <ulid: 01FXPKR2PK2SABN058FTYN5AD1, mint: 1646611200003, maxt: 1646784000000, range: 47h59m59s>, <ulid: 01FXK57ESVZCPZF6QS52G3JXDM, mint: 1646640000008, maxt: 1646668800000, range: 7h59m59s>\n[mint: 1646668800027, maxt: 1646697600000, range: 7h59m59s, blocks: 2]: <ulid: 01FXPKR2PK2SABN058FTYN5AD1, mint: 1646611200003, maxt: 1646784000000, range: 47h59m59s>, <ulid: 01FXM0KF40TSGYXWB2PJKD4D26, mint: 1646668800027, maxt: 1646697600000, range: 7h59m59s>\n[mint: 1646755200010, maxt: 1646762400000, range: 1h59m59s, blocks: 3]: <ulid: 01FXPKR2PK2SABN058FTYN5AD1, mint: 1646611200003, maxt: 1646784000000, range: 47h59m59s>, <ulid: 01FXPJW1QC5J2ER4D9GCC55S1K, mint: 1646755200010, maxt: 1646784000000, range: 7h59m59s>, <ulid: 01FXNEE21BKQD64NF22B42YCAD, mint: 1646755200010, maxt: 1646762400000, range: 1h59m59s>\n[mint: 1646762400026, maxt: 1646769600000, range: 1h59m59s, blocks: 3]: <ulid: 01FXPKR2PK2SABN058FTYN5AD1, mint: 1646611200003, maxt: 1646784000000, range: 47h59m59s>, <ulid: 01FXPJW1QC5J2ER4D9GCC55S1K, mint: 1646755200010, maxt: 1646784000000, range: 7h59m59s>, <ulid: 01FXNN9S94T43N1JYQWTTKMJ6D, mint: 1646762400026, maxt: 1646769600000, range: 1h59m59s>\n[mint: 1646798400008, maxt: 1646805600000, range: 1h59m59s, blocks: 2]: <ulid: 01FXQEP1YAB9GP2AXN5BEGXR1Q, mint: 1646784000020, maxt: 1646812800000, range: 7h59m59s>, <ulid: 01FXPQMDH4CSEKRM8XNFT8JABK, mint: 1646798400008, maxt: 1646805600000, range: 1h59m59s>\n[mint: 1646827200000, maxt: 1646834400000, range: 2h0m0s, blocks: 2]: <ulid: 01FXR9QTH8ZHMAEH0GK5FGEJ1F, mint: 1646812800009, maxt: 1646841600000, range: 7h59m59s>, <ulid: 01FXQK3AH3AA2GWSES78SZ19PC, mint: 1646827200000, maxt: 1646834400000, range: 2h0m0s>\n[mint: 1646733600036, maxt: 1646740800000, range: 1h59m59s, blocks: 3]: <ulid: 01FXPKR2PK2SABN058FTYN5AD1, mint: 1646611200003, maxt: 1646784000000, range: 47h59m59s>, <ulid: 01FXNQ98PXRBDX8E18ZE5Q0E62, mint: 1646726400010, maxt: 1646755200000, range: 7h59m59s>, <ulid: 01FXMSTW93GF0K65F0A5W5CZYK, mint: 1646733600036, maxt: 1646740800000, range: 1h59m59s>\n[mint: 1646812800009, maxt: 1646820000000, range: 1h59m59s, blocks: 2]: <ulid: 01FXQ5BW1497XB6RY72HKKDBSV, mint: 1646812800009, maxt: 1646820000000, range: 1h59m59s>, <ulid: 01FXR9QTH8ZHMAEH0GK5FGEJ1F, mint: 1646812800009, maxt: 1646841600000, range: 7h59m59s>\n[mint: 1646820000011, maxt: 1646827200000, range: 1h59m59s, blocks: 2]: <ulid: 01FXR9QTH8ZHMAEH0GK5FGEJ1F, mint: 1646812800009, maxt: 1646841600000, range: 7h59m59s>, <ulid: 01FXQC7KATFN7306DPFB8RVSSA, mint: 1646820000011, maxt: 1646827200000, range: 1h59m59s>"

[AndrewNiven1]

Thanks for the quick fix!