Thanos Compactor : retention.resolution-1h data are deleted instead of downsampled #5664
Comments
|
sounds like same issue as i'm having, discussed in #5435 (comment) |
|
I replied to that issue #5435 (comment). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Thanos, Prometheus and Golang version used:
thanos, version 0.26.0 (branch: HEAD, revision: 17c5764)
build user: root@95d2db822102
build date: 20220505-12:53:36
go version: go1.17.9
platform: linux/amd64
Object Storage Provider:
FILESYSTEM
What happened:
We have the following thanos compact configuration (on Server named A):
--data-dir=/mnt/thanos_tsdb/thanos-compacter
--objstore.config-file=/opt/thanos/bucket_config.yaml
--retention.resolution-raw=2d
--retention.resolution-5m=10d
--retention.resolution-1h=365d
--wait --wait-interval=5m
--http-address=0.0.0.0:10902
--compact.concurrency=24
--log.level=debug
--delete-delay=48h
--deduplication.replica-label=replica
We can't keep data older than 11 days (for example today, August 31th, we don't have data older than August 20th)
What you expected to happen:
We would like to get data store for 1 year with retention: retention.resolution-1h
How to reproduce it (as minimally and precisely as possible):
We have just configured 2 prometheus servers with thanos sidecar (called server B and C for high availability).
Here configuration for server B (same on server C)
Prometheus config:
--config.file /etc/prometheus/prometheus.yml
--storage.tsdb.max-block-duration=2h
--storage.tsdb.min-block-duration=2h
--storage.tsdb.retention.time=4d
--storage.tsdb.path=/opt/prometheus/data
--web.console.libraries=/etc/prometheus/console_libraries
--web.console.templates=/etc/prometheus/consoles
--web.listen-address=0.0.0.0:9090
--web.enable-lifecycle
--web.enable-admin-api
Sidecar config:
--log.level=info
--log.format=logfmt
--tsdb.path=/opt/prometheus/data
--objstore.config-file=/opt/thanos/sidecar_config.yaml
--prometheus.url=http://localhost:9090
--http-address=0.0.0.0:19191
--grpc-address=0.0.0.0:19090
Content of objstore file :
type: FILESYSTEM
config:
directory: "/mnt/thanos_tsdb/tsdb"
We are using a NFS mounting point between Prometheus and Thanos servers
Full logs to relevant components:
Here is the log returned by command :
/opt/thanos/current/thanos tools bucket inspect --objstore.config-file /opt/thanos/bucket_config.yaml
level=info ts=2022-08-31T21:04:38.053064409Z caller=factory.go:49 msg="loading bucket configuration" level=info ts=2022-08-31T21:04:38.098541041Z caller=fetcher.go:470 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=44.373798ms duration_ms=44 cached=99 returned=99 partial=0 | ULID | FROM | UNTIL | RANGE | UNTIL-DOWN | #SERIES | #SAMPLES | #CHUNKS | COMP-LEVEL | COMP-FAILED | LABELS | RESOLUTION | SOURCE | |----------------------------|----------------------|----------------------|----------------|-------------|------------|----------------|-------------|------------|-------------|---------------------|------------|-----------| | 01G568T1H9WFBCDZ8HS1T5RXJ4 | 2021-11-11T00:00:00Z | 2021-11-25T00:00:00Z | 335h59m59.998s | - | 625,589 | 81,343 | 1,315,357 | 5 | false | env=infra | 1h0m0s | compactor | | 01G1MWGP15VYWJM4NHSQMRXTPC | 2022-01-06T00:00:00Z | 2022-01-20T00:00:00Z | 335h59m59.72s | - | 538,998 | 18,205,645 | 650,824 | 4 | false | env=infra,replica=a | 1h0m0s | compactor | | 01GAX0MY7XNS0KTA43TXKDJW9X | 2022-08-18T00:00:00Z | 2022-08-20T00:00:00Z | 48h0m0s | 192h0m0s | 13,570,802 | 1,515,524,728 | 22,499,179 | 5 | false | env=infra | 5m0s | compactor | | 01GB26HSY9Q65NKS2R185CS5D3 | 2022-08-20T00:00:00Z | 2022-08-22T00:00:00Z | 48h0m0s | 192h0m0s | 14,101,725 | 1,541,095,562 | 23,126,972 | 4 | false | env=infra | 5m0s | compactor | | 01GB7BEDFJ1XWBVCDNDDA9KWQR | 2022-08-22T00:00:00Z | 2022-08-24T00:00:00Z | 48h0m0s | 192h0m0s | 14,846,398 | 1,546,176,203 | 23,884,477 | 4 | false | env=infra | 5m0s | compactor | | 01GBCF48PEC4GP07G4JZGPNQSM | 2022-08-24T00:00:00Z | 2022-08-26T00:00:00Z | 48h0m0s | 192h0m0s | 15,016,473 | 1,547,680,420 | 24,082,000 | 4 | false | env=infra | 5m0s | compactor | | 01GBHD1F140972AXJE1CYAVXYN | 2022-08-26T00:00:00Z | 2022-08-28T00:00:00Z | 48h0m0s | -8h0m0s | 15,012,229 | 44,980,125,506 | 394,468,222 | 4 | false | env=infra | 0s | compactor | | 01GBHNQ4WTH3Q3ETWYMKEPNBHM | 2022-08-26T00:00:00Z | 2022-08-28T00:00:00Z | 48h0m0s | 192h0m0s | 15,011,291 | 1,572,096,810 | 24,259,854 | 4 | false | env=infra | 5m0s | compactor | | 01GBJ4QK9Q3ZFH9KPSX7QNYS72 | 2022-08-28T00:00:00Z | 2022-08-28T08:00:00Z | 8h0m0s | 32h0m0s | 4,892,252 | 7,536,324,624 | 66,055,898 | 3 | false | env=infra | 0s | compactor | | 01GBPHR2AHDFHJ69A8DF97K2YZ | 2022-08-28T00:00:00Z | 2022-08-30T00:00:00Z | 48h0m0s | -8h0m0s | 16,036,333 | 45,272,349,735 | 398,186,381 | 4 | false | env=infra | 0s | compactor | | 01GBPTF4AEB7XRX7CVDS3Y0GWM | 2022-08-28T00:00:00Z | 2022-08-30T00:00:00Z | 48h0m0s | 192h0m0s | 16,035,488 | 1,583,682,834 | 25,342,535 | 4 | false | env=infra | 5m0s | compactor | | 01GBK7590N211X6QW5BE025YTW | 2022-08-28T08:00:00Z | 2022-08-28T16:00:00Z | 7h59m59.989s | 32h0m0.011s | 4,880,026 | 7,528,623,463 | 66,000,218 | 3 | false | env=infra | 0s | compactor | | 01GBM2M89ETAKM34KC53QSK69Z | 2022-08-28T16:00:00Z | 2022-08-29T00:00:00Z | 7h59m59.999s | 32h0m0.001s | 4,883,703 | 7,537,505,951 | 66,049,466 | 3 | false | env=infra | 0s | compactor | | 01GBMQ4TA7YT23PWHHTCBFX9N6 | 2022-08-29T00:00:00Z | 2022-08-29T08:00:00Z | 8h0m0s | 32h0m0s | 4,968,250 | 7,541,032,777 | 66,156,894 | 3 | false | env=infra | 0s | compactor | | 01GBMP2GN7WBJ1AS3Y9XT4CQMM | 2022-08-29T08:00:00Z | 2022-08-29T10:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,459,395 | 1,883,932,902 | 16,687,322 | 2 | false | env=infra | 0s | compactor | | 01GBNSJTT04QAA338M8Q8GP3JH | 2022-08-29T08:00:00Z | 2022-08-29T16:00:00Z | 7h59m59.999s | 32h0m0.001s | 5,519,118 | 7,621,867,400 | 67,501,138 | 3 | false | env=infra | 0s | compactor | | 01GBMWY8QTKFHNGAJ07ZJ102W1 | 2022-08-29T10:00:00Z | 2022-08-29T12:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,445,342 | 1,880,107,271 | 16,638,204 | 2 | false | env=infra | 0s | compactor | | 01GBN3T6GPSJXWGWR5E5R9M0A6 | 2022-08-29T12:00:00Z | 2022-08-29T14:00:00Z | 2h0m0s | 38h0m0s | 3,508,101 | 1,936,380,112 | 17,146,122 | 2 | false | env=infra | 0s | compactor | | 01GBNANYKDGH25ANGRQSQJ2A2P | 2022-08-29T14:00:00Z | 2022-08-29T16:00:00Z | 2h0m0s | 38h0m0s | 3,509,602 | 1,921,447,115 | 17,029,490 | 2 | false | env=infra | 0s | compactor | | 01GBNHHHVKBR13RE0G25AN4QCZ | 2022-08-29T16:00:00Z | 2022-08-29T18:00:00Z | 2h0m0s | 38h0m0s | 3,447,151 | 1,881,213,593 | 16,643,608 | 2 | false | env=infra | 0s | compactor | | 01GBPE385JJPBPGMTBFJXFBDMM | 2022-08-29T16:00:00Z | 2022-08-30T00:00:00Z | 8h0m0s | 32h0m0s | 5,445,243 | 7,506,995,520 | 66,422,767 | 3 | false | env=infra | 0s | compactor | | 01GBNPEV4RF4XPPAQYXZYQMHYQ | 2022-08-29T18:00:00Z | 2022-08-29T20:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,434,503 | 938,394,598 | 9,075,163 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBNPEV5DW3JKKQFVQ604719Y | 2022-08-29T18:00:00Z | 2022-08-29T20:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,434,269 | 940,535,283 | 9,085,934 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBNRDCESGZQXKVWJQD0ZYGEP | 2022-08-29T18:00:00Z | 2022-08-29T20:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,434,940 | 1,878,193,310 | 16,617,231 | 2 | false | env=infra | 0s | compactor | | 01GBNXAJCQVZ3XR952A7N3PV46 | 2022-08-29T20:00:00Z | 2022-08-29T22:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,441,579 | 935,972,808 | 9,061,668 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBNXAJD48R8Q661763X8VN3J | 2022-08-29T20:00:00Z | 2022-08-29T22:00:00Z | 2h0m0s | 38h0m0s | 3,441,861 | 938,625,817 | 9,082,562 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBNZ8Z6567ZCN7Q97AZ0BHZJ | 2022-08-29T20:00:00Z | 2022-08-29T22:00:00Z | 2h0m0s | 38h0m0s | 3,442,404 | 1,873,852,185 | 16,579,199 | 2 | false | env=infra | 0s | compactor | | 01GBP469MPG5T3X763W160R902 | 2022-08-29T22:00:00Z | 2022-08-30T00:00:00Z | 2h0m0s | 38h0m0s | 3,443,937 | 937,228,306 | 9,062,752 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBP469NC44KGK4FE8YT34RRD | 2022-08-29T22:00:00Z | 2022-08-30T00:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,443,770 | 937,266,879 | 9,064,075 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBP64NHFQ0Y58RFA8EHEH5WP | 2022-08-29T22:00:00Z | 2022-08-30T00:00:00Z | 2h0m0s | 38h0m0s | 3,444,554 | 1,873,736,432 | 16,582,729 | 2 | false | env=infra | 0s | compactor | | 01GBPB20WPYB6ASJ08A362RV09 | 2022-08-30T00:00:00Z | 2022-08-30T02:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,445,152 | 940,763,763 | 9,095,184 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBPB20XETRAJZBRW76C20N02 | 2022-08-30T00:00:00Z | 2022-08-30T02:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,445,027 | 940,792,839 | 9,082,585 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBPD0PQR24XCS734TS37F52P | 2022-08-30T00:00:00Z | 2022-08-30T02:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,445,836 | 1,880,774,281 | 16,641,461 | 2 | false | env=infra | 0s | compactor | | 01GBQGG5ZKAHR1D5YKYCQ3WWHZ | 2022-08-30T00:00:00Z | 2022-08-30T08:00:00Z | 7h59m59.999s | 32h0m0.001s | 5,467,235 | 7,521,869,973 | 66,556,519 | 3 | false | env=infra | 0s | compactor | | 01GBPHXR4NPW4GP6SZCHA869BD | 2022-08-30T02:00:00Z | 2022-08-30T04:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,442,824 | 940,831,140 | 9,086,163 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBPHXR5C9B41AJYTTHJBZWYN | 2022-08-30T02:00:00Z | 2022-08-30T04:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,441,743 | 940,833,160 | 9,085,150 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBPQ9APZ8GXMSP057760MHRC | 2022-08-30T02:00:00Z | 2022-08-30T04:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,443,411 | 1,880,867,162 | 16,639,458 | 2 | false | env=infra | 0s | compactor | | 01GBPRSFCQNMS6M8RMDZ9W42C9 | 2022-08-30T04:00:00Z | 2022-08-30T06:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,446,070 | 940,575,802 | 9,096,686 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBPRSFDCY17FQQCRFSRHRAR2 | 2022-08-30T04:00:00Z | 2022-08-30T06:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,446,207 | 940,518,067 | 9,094,976 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBPXZ172Z42CFMEYA0PTG8BV | 2022-08-30T04:00:00Z | 2022-08-30T06:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,446,683 | 1,880,284,060 | 16,642,217 | 2 | false | env=infra | 0s | compactor | | 01GBPZN6MT873RQK9X3M2D61NY | 2022-08-30T06:00:00Z | 2022-08-30T08:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,435,781 | 940,330,201 | 9,089,981 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBPZN6NDWQNS37EP6Z2PYC3Y | 2022-08-30T06:00:00Z | 2022-08-30T08:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,435,948 | 940,375,909 | 9,090,128 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBQ1KRK0DEY6MX5Z84W3GESR | 2022-08-30T06:00:00Z | 2022-08-30T08:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,436,606 | 1,879,944,470 | 16,633,383 | 2 | false | env=infra | 0s | compactor | | 01GBQ6GXWSJ0QFKQEDGX025WBS | 2022-08-30T08:00:00Z | 2022-08-30T10:00:00Z | 2h0m0s | 38h0m0s | 3,445,634 | 940,928,457 | 9,101,133 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBQ6GXXGP8ZA6BAE4JRC25Q4 | 2022-08-30T08:00:00Z | 2022-08-30T10:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,445,666 | 941,043,074 | 9,101,154 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBQ8FBX9Z95D7VEAF113ADDN | 2022-08-30T08:00:00Z | 2022-08-30T10:00:00Z | 2h0m0s | 38h0m0s | 3,446,268 | 1,881,196,897 | 16,644,744 | 2 | false | env=infra | 0s | compactor | | 01GBR546E1PC9WZZ8FMBY5Y7D7 | 2022-08-30T08:00:00Z | 2022-08-30T16:00:00Z | 8h0m0s | 32h0m0s | 5,453,571 | 7,527,298,450 | 66,601,377 | 3 | false | env=infra | 0s | compactor | | 01GBQDCN4QFHRF30PZ2PPG8EBX | 2022-08-30T10:00:00Z | 2022-08-30T12:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,437,661 | 940,478,535 | 9,047,171 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBQDCN5CA4GRCZX8QGRWBF2Z | 2022-08-30T10:00:00Z | 2022-08-30T12:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,437,532 | 940,871,840 | 9,093,744 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBQFB4KZBW65EKSWBSWYZTCS | 2022-08-30T10:00:00Z | 2022-08-30T12:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,438,158 | 1,880,590,425 | 16,643,744 | 2 | false | env=infra | 0s | compactor | | 01GBQM8CCRV4KE3Y0FG4RY6760 | 2022-08-30T12:00:00Z | 2022-08-30T14:00:00Z | 2h0m0s | 38h0m0s | 3,452,715 | 942,896,928 | 9,075,314 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBQM8CDC22ZFDR6K0V6KQ4FF | 2022-08-30T12:00:00Z | 2022-08-30T14:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,452,804 | 943,298,117 | 9,122,010 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBQP6VWMS2F232FQE6R80YBA | 2022-08-30T12:00:00Z | 2022-08-30T14:00:00Z | 2h0m0s | 38h0m0s | 3,453,358 | 1,885,436,927 | 16,683,937 | 2 | false | env=infra | 0s | compactor | | 01GBQV43MPATFC3ZYCSQVXY5CS | 2022-08-30T14:00:00Z | 2022-08-30T16:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,439,485 | 940,329,575 | 9,043,484 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBQV43ND7Z63FV7SR3WVKBWX | 2022-08-30T14:00:00Z | 2022-08-30T16:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,439,435 | 940,520,980 | 9,090,112 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBQX2HCAXD83ZTBZJC23M1WS | 2022-08-30T14:00:00Z | 2022-08-30T16:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,440,231 | 1,880,074,201 | 16,628,952 | 2 | false | env=infra | 0s | compactor | | 01GBR1ZTX5Z73X0PJF8043YB8E | 2022-08-30T16:00:00Z | 2022-08-30T18:00:00Z | 2h0m0s | 38h0m0s | 3,441,509 | 940,563,846 | 9,092,565 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBR1ZTXCWEQFQV8KZB9TFVM0 | 2022-08-30T16:00:00Z | 2022-08-30T18:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,442,450 | 940,598,847 | 9,093,491 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBR3YBYQER3FXA0M2TQ1RTD2 | 2022-08-30T16:00:00Z | 2022-08-30T18:00:00Z | 2h0m0s | 38h0m0s | 3,443,110 | 1,880,382,114 | 16,632,309 | 2 | false | env=infra | 0s | compactor | | 01GBS0JSABYXR1KVY75SJ9JBET | 2022-08-30T16:00:00Z | 2022-08-31T00:00:00Z | 8h0m0s | 32h0m0s | 5,432,577 | 7,518,557,570 | 66,497,871 | 3 | false | env=infra | 0s | compactor | | 01GBR8VJ4QS6HYBPA7HVDEWDPQ | 2022-08-30T18:00:00Z | 2022-08-30T20:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,428,773 | 939,973,172 | 9,066,967 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBR8VJ5K6V67G6EWFT24CQTP | 2022-08-30T18:00:00Z | 2022-08-30T20:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,428,913 | 940,095,144 | 9,067,132 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBRAT4PK2RXESYNB3AZJSKVX | 2022-08-30T18:00:00Z | 2022-08-30T20:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,429,394 | 1,879,296,347 | 16,615,450 | 2 | false | env=infra | 0s | compactor | | 01GBRFQ9CQXYECRQPB9FBKXF8M | 2022-08-30T20:00:00Z | 2022-08-30T22:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,437,352 | 939,839,001 | 9,075,606 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBRFQ9DFNWZSAEP310HK602F | 2022-08-30T20:00:00Z | 2022-08-30T22:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,437,373 | 940,162,154 | 9,075,588 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBRHNWCTZ38ABNMP6R7NTPV7 | 2022-08-30T20:00:00Z | 2022-08-30T22:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,437,900 | 1,879,248,380 | 16,626,309 | 2 | false | env=infra | 0s | compactor | | 01GBRPK0MQF03ASD1TQ8WX7SAF | 2022-08-30T22:00:00Z | 2022-08-31T00:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,436,272 | 940,059,212 | 9,040,414 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBRPK0NFDWQ0DBJ45FY33NJR | 2022-08-30T22:00:00Z | 2022-08-31T00:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,436,105 | 940,326,123 | 9,086,996 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBRRHGP3XJ7TMMH7TVEEYG1T | 2022-08-30T22:00:00Z | 2022-08-31T00:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,436,856 | 1,879,630,729 | 16,623,803 | 2 | false | env=infra | 0s | compactor | | 01GBRXEQWQ9DMAZY2Z4EP3FGHZ | 2022-08-31T00:00:00Z | 2022-08-31T02:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,439,937 | 940,241,920 | 9,091,601 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBRXEQXDZY43R3D35PE7A8CT | 2022-08-31T00:00:00Z | 2022-08-31T02:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,439,985 | 940,329,010 | 9,091,673 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBRZDKK0Z8C68XNE9RHZ27D5 | 2022-08-31T00:00:00Z | 2022-08-31T02:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,440,594 | 1,879,797,779 | 16,629,512 | 2 | false | env=infra | 0s | compactor | | 01GBT2W2K3P09FBZDB74J939GM | 2022-08-31T00:00:00Z | 2022-08-31T08:00:00Z | 7h59m59.999s | 32h0m0.001s | 5,462,767 | 7,518,101,037 | 66,519,085 | 3 | false | env=infra | 0s | compactor | | 01GBS4AF4RY85ZJQ8KXQ3KCAKF | 2022-08-31T02:00:00Z | 2022-08-31T04:00:00Z | 2h0m0s | 38h0m0s | 3,433,401 | 940,430,455 | 9,074,041 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBS4AF5C6A474APAZ2MN7Q21 | 2022-08-31T02:00:00Z | 2022-08-31T04:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,433,496 | 940,424,855 | 9,073,780 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBS68TA2BEG97M7V6JXBTKPY | 2022-08-31T02:00:00Z | 2022-08-31T04:00:00Z | 2h0m0s | 38h0m0s | 3,433,964 | 1,880,076,396 | 16,623,977 | 2 | false | env=infra | 0s | compactor | | 01GBSB66CRRSPEMP4V9X9BQTCX | 2022-08-31T04:00:00Z | 2022-08-31T06:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,444,329 | 940,261,349 | 9,095,702 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBSB66D5TB7SHPNP9B8NQ7B2 | 2022-08-31T04:00:00Z | 2022-08-31T06:00:00Z | 2h0m0s | 38h0m0s | 3,444,303 | 940,633,485 | 9,093,838 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBSD4Q3WSGQ7X106QSX83E8C | 2022-08-31T04:00:00Z | 2022-08-31T06:00:00Z | 2h0m0s | 38h0m0s | 3,444,937 | 1,880,125,385 | 16,638,953 | 2 | false | env=infra | 0s | compactor | | 01GBSJ1XMPMAHGBJW7ER0JZXPJ | 2022-08-31T06:00:00Z | 2022-08-31T08:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,434,025 | 938,242,293 | 9,068,309 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBSJ1XNCJT63CM13Y9K0GM9N | 2022-08-31T06:00:00Z | 2022-08-31T08:00:00Z | 1h59m59.963s | 38h0m0.037s | 3,434,443 | 940,592,283 | 9,090,849 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBSM0BQ2EAA65CXBP1EW88T1 | 2022-08-31T06:00:00Z | 2022-08-31T08:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,435,032 | 1,878,101,477 | 16,626,643 | 2 | false | env=infra | 0s | compactor | | 01GBSRXMWR0JQFB833CWYBABV6 | 2022-08-31T08:00:00Z | 2022-08-31T10:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,444,399 | 940,506,703 | 9,094,957 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBSRXMXDH5FHF9Q19H63SJG5 | 2022-08-31T08:00:00Z | 2022-08-31T10:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,444,065 | 940,505,293 | 9,094,620 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBSTW00YZB6YSPZGFVS0011Y | 2022-08-31T08:00:00Z | 2022-08-31T10:00:00Z | 1h59m59.999s | 38h0m0.001s | 3,444,978 | 1,880,206,556 | 16,651,169 | 2 | false | env=infra | 0s | compactor | | 01GBSZSC579PPH3AWPPDPNHX7H | 2022-08-31T10:00:00Z | 2022-08-31T12:00:00Z | 1h59m59.992s | 38h0m0.008s | 3,432,331 | 941,565,840 | 9,091,068 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBSZSC5C7QSFDX9MG57KE9M8 | 2022-08-31T10:00:00Z | 2022-08-31T12:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,432,296 | 941,009,607 | 9,091,012 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBT1QR810N8RQX0VC9G0D3FR | 2022-08-31T10:00:00Z | 2022-08-31T12:00:00Z | 1h59m59.992s | 38h0m0.008s | 3,432,901 | 1,881,774,233 | 16,639,299 | 2 | false | env=infra | 0s | compactor | | 01GBT6N3DCBA5T7PTNQX3EQZ8E | 2022-08-31T12:00:00Z | 2022-08-31T14:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,445,266 | 940,071,423 | 9,102,024 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBT6N3DKRR7JRYDRKN2WE849 | 2022-08-31T12:00:00Z | 2022-08-31T14:00:00Z | 2h0m0s | 38h0m0s | 3,270,580 | 702,405,532 | 7,834,390 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBT95PPH8A6S5ZC4CNVJRTVZ | 2022-08-31T12:00:00Z | 2022-08-31T14:00:00Z | 2h0m0s | 38h0m0s | 3,447,217 | 1,641,921,183 | 15,554,656 | 2 | false | env=infra | 0s | compactor | | 01GBTDGTNE4JC1SXPK1M3JKDEH | 2022-08-31T14:00:00Z | 2022-08-31T16:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,434,867 | 941,490,510 | 9,090,022 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBTFF5PXZ2T6F4X4P2TSQFHM | 2022-08-31T14:00:00Z | 2022-08-31T16:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,435,622 | 1,833,347,374 | 16,641,159 | 2 | false | env=infra | 0s | compactor | | 01GBTDMVJYYXE2CHS6F1FRKPC5 | 2022-08-31T14:02:12Z | 2022-08-31T16:00:00Z | 1h57m47.99s | 38h2m12.01s | 3,398,445 | 892,564,760 | 7,169,683 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBTMCHXDYCW41HXTHC8SF3MT | 2022-08-31T16:00:00Z | 2022-08-31T18:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,441,890 | 941,840,072 | 9,100,690 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBTMCHXHYG7TSW86DC7CEV31 | 2022-08-31T16:00:00Z | 2022-08-31T18:00:00Z | 2h0m0s | 38h0m0s | 3,441,727 | 941,169,776 | 9,053,750 | 1 | false | env=infra,replica=b | 0s | sidecar | | 01GBTPAZ77WY7FM5Q14V662JQ7 | 2022-08-31T16:00:00Z | 2022-08-31T18:00:00Z | 2h0m0s | 38h0m0s | 3,442,370 | 1,882,264,911 | 16,649,455 | 2 | false | env=infra | 0s | compactor | | 01GBTV895D8R8DX0CPTBBPPYCG | 2022-08-31T18:00:00Z | 2022-08-31T20:00:00Z | 1h59m59.97s | 38h0m0.03s | 3,430,618 | 941,452,874 | 9,088,818 | 1 | false | env=infra,replica=a | 0s | sidecar | | 01GBTV895NV8052JSZFV984BKJ | 2022-08-31T18:00:00Z | 2022-08-31T20:00:00Z | 2h0m0s | 38h0m0s | 3,430,409 | 941,453,443 | 9,088,726 | 1 | false | env=infra,replica=b | 0s | sidecar | level=info ts=2022-08-31T21:04:38.132890675Z caller=main.go:161 msg=exiting
Here are a part of the logs of compactor :
level=info ts=2022-07-11T08:02:21.823519062Z caller=blocks_cleaner.go:44 msg="started cleaning of blocks marked for deletion" level=debug ts=2022-07-11T08:02:22.524945844Z caller=block.go:250 msg="deleted file" file=01G7CM0TXC0133RJZVNQMCGKBN/deletion-mark.json bucket="tracing: fs: /mnt/thanos_tsdb/tsdb" level=info ts=2022-07-11T08:02:22.525008244Z caller=blocks_cleaner.go:54 msg="deleted block marked for deletion" block=01G7CM0TXC0133RJZVNQMCGKBN level=debug ts=2022-07-11T08:02:22.874884141Z caller=block.go:250 msg="deleted file" file=01G7FNS8X9WQ837TMJ6M9GV0JY/deletion-mark.json bucket="tracing: fs: /mnt/thanos_tsdb/tsdb" level=info ts=2022-07-11T08:02:22.87495726Z caller=blocks_cleaner.go:54 msg="deleted block marked for deletion" block=01G7FNS8X9WQ837TMJ6M9GV0JY level=debug ts=2022-07-11T08:02:23.404241585Z caller=block.go:250 msg="deleted file" file=01G7E5QEH42N773CQAPZNQCX6Z/deletion-mark.json bucket="tracing: fs: /mnt/thanos_tsdb/tsdb" level=info ts=2022-07-11T08:02:23.404326121Z caller=blocks_cleaner.go:54 msg="deleted block marked for deletion" block=01G7E5QEH42N773CQAPZNQCX6Z level=debug ts=2022-07-11T08:02:24.225393492Z caller=block.go:250 msg="deleted file" file=01G7GR2PWGWH1F41BRMRT2V55Z/deletion-mark.json bucket="tracing: fs: /mnt/thanos_tsdb/tsdb" level=info ts=2022-07-11T08:02:24.225470301Z caller=blocks_cleaner.go:54 msg="deleted block marked for deletion" block=01G7GR2PWGWH1F41BRMRT2V55Z level=debug ts=2022-07-11T08:02:24.912591077Z caller=block.go:250 msg="deleted file" file=01G7DA8SZ92CGJ92ZFB5KNYQ8W/deletion-mark.json bucket="tracing: fs: /mnt/thanos_tsdb/tsdb" level=info ts=2022-07-11T08:02:24.912643498Z caller=blocks_cleaner.go:54 msg="deleted block marked for deletion" block=01G7DA8SZ92CGJ92ZFB5KNYQ8W level=debug ts=2022-07-11T08:02:30.12934779Z caller=block.go:250 msg="deleted file" file=01G7F16FCNANZ8YW1NBWEWZN67/deletion-mark.json bucket="tracing: fs: /mnt/thanos_tsdb/tsdb" level=info ts=2022-07-11T08:02:30.129386683Z caller=blocks_cleaner.go:54 msg="deleted block marked for deletion" block=01G7F16FCNANZ8YW1NBWEWZN67
Anything else we need to know:
When I check blocks with Thanos WebUI I got this picture where I don't see any blocks for the retention : --retention.resolution-1h=365d
Environment:
uname -a): Linux 4.18.0-394.el8.x86_64 x86_64 x86_64 x86_64 GNU/LinuxThanks for your help
Regards
Ben
The text was updated successfully, but these errors were encountered: