Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compact: shutdown unexpectedly #2287

Closed
Gnoale opened this issue Mar 18, 2020 · 2 comments
Closed

compact: shutdown unexpectedly #2287

Gnoale opened this issue Mar 18, 2020 · 2 comments

Comments

@Gnoale
Copy link

Gnoale commented Mar 18, 2020

Thanos, Prometheus and Golang version used:
thanos, version 0.11.0
go1.13.6
prometheus, version 2.6.0

Object Storage Provider:
S3

What happened:
I started the compact service on a VM on which the store service is already running.
It went well during about 2 hours until the compact module stopped suddenly working.

Now the compact service just won't keep running anymore, the working directory is empty and from the logs it seems everything is fine until some exception from http.go is raised
level=info ts=2020-03-18T13:48:12.910721661Z caller=http.go:81 service=http/server component=compact msg="internal server shutdown" err=null

What you expected to happen:
Compact should not stop running

How to reproduce it (as minimally and precisely as possible):
Run the compact and store services in the same environment (operating system, users) against the same s3 storage object provider

Full logs to relevant components:

Logs from the service at the point where it stopped working (started Mar 18 08:54:14 stopped Mar 18 10:25:28

Mar 18 08:54:14 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:54:14.256484336Z caller=main.go:152 msg="Tracing will be disabled"
Mar 18 08:54:14 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:54:14.256603659Z caller=factory.go:46 msg="loading bucket configuration"
Mar 18 08:54:14 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:54:14.25713095Z caller=compact.go:386 msg="starting compact node"
Mar 18 08:54:14 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:54:14.257157028Z caller=intrumentation.go:52 msg="changing probe status" status=ready
Mar 18 08:54:14 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:54:14.257425848Z caller=compact.go:858 msg="start sync of metas"
Mar 18 08:54:14 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:54:14.258312265Z caller=intrumentation.go:64 msg="changing probe status" status=healthy
Mar 18 08:54:14 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:54:14.258507844Z caller=http.go:56 service=http/server component=compact msg="listening for r
Mar 18 08:54:15 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:54:15.447950177Z caller=fetcher.go:368 component=block.MetaFetcher msg="successfully fetched 
Mar 18 08:54:15 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:54:15.448009153Z caller=compact.go:864 msg="start of GC"
Mar 18 08:54:15 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:54:15.448020436Z caller=compact.go:872 msg="start of compaction"
Mar 18 08:54:17 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:54:17.528335715Z caller=compact.go:441 msg="compact blocks" count=3 mint=1583834400000 maxt=1
Mar 18 08:54:19 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:54:19.470655716Z caller=compact.go:763 compactionGroup=0@14539196864129613008 msg="deleting c
Mar 18 08:54:20 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:54:20.416941362Z caller=compact.go:763 compactionGroup=0@14539196864129613008 msg="deleting c
Mar 18 08:54:21 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:54:21.246485865Z caller=compact.go:763 compactionGroup=0@14539196864129613008 msg="deleting c
Mar 18 08:54:41 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:54:41.005218399Z caller=compact.go:441 msg="compact blocks" count=3 mint=1583839044595 maxt=1
Mar 18 08:54:47 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:54:47.221800803Z caller=compact.go:763 compactionGroup=0@15016429373970399229 msg="deleting c
Mar 18 08:54:48 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:54:48.131118347Z caller=compact.go:763 compactionGroup=0@15016429373970399229 msg="deleting c
Mar 18 08:54:49 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:54:49.278267016Z caller=compact.go:763 compactionGroup=0@15016429373970399229 msg="deleting c
Mar 18 08:55:13 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:55:13.408279073Z caller=compact.go:441 msg="compact blocks" count=4 mint=1583831785464 maxt=1
Mar 18 08:55:19 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:55:19.397433332Z caller=compact.go:763 compactionGroup=0@3010882595722085081 msg="deleting co
Mar 18 08:55:20 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:55:20.695568834Z caller=compact.go:763 compactionGroup=0@3010882595722085081 msg="deleting co
Mar 18 08:55:21 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:55:21.632591564Z caller=compact.go:763 compactionGroup=0@3010882595722085081 msg="deleting co
Mar 18 08:55:22 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:55:22.741652735Z caller=compact.go:763 compactionGroup=0@3010882595722085081 msg="deleting co
Mar 18 08:55:23 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:55:23.755750835Z caller=compact.go:858 msg="start sync of metas"
Mar 18 08:55:24 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:55:24.314209977Z caller=fetcher.go:368 component=block.MetaFetcher msg="successfully fetched 
Mar 18 08:55:24 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:55:24.314259974Z caller=compact.go:864 msg="start of GC"
Mar 18 08:55:24 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:55:24.31427172Z caller=compact.go:872 msg="start of compaction"
Mar 18 08:55:25 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:55:25.631922873Z caller=compact.go:441 msg="compact blocks" count=2 mint=1583856000000 maxt=1
Mar 18 08:55:27 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:55:27.347431223Z caller=compact.go:763 compactionGroup=0@14539196864129613008 msg="deleting c
Mar 18 08:55:28 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:55:28.362868279Z caller=compact.go:763 compactionGroup=0@14539196864129613008 msg="deleting c
Mar 18 08:56:11 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:56:11.686405402Z caller=compact.go:441 msg="compact blocks" count=4 mint=1583856000000 maxt=1
Mar 18 08:56:22 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:56:22.169351657Z caller=compact.go:763 compactionGroup=0@15016429373970399229 msg="deleting c
Mar 18 08:56:23 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:56:23.272697635Z caller=compact.go:763 compactionGroup=0@15016429373970399229 msg="deleting c
Mar 18 08:56:24 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:56:24.589822719Z caller=compact.go:763 compactionGroup=0@15016429373970399229 msg="deleting c
Mar 18 08:56:25 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:56:25.814070766Z caller=compact.go:763 compactionGroup=0@15016429373970399229 msg="deleting c
Mar 18 08:57:07 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T08:57:07.892248327Z caller=compact.go:441 msg="compact blocks" count=4 mint=1583856000000 maxt=1
....
....
....
Mar 18 10:20:52 thanos-store.kayrros thanos[31070]: level=warn ts=2020-03-18T10:20:52.08481167Z caller=streamed_block_writer.go:115 msg="empty chunks happened, skip series" 
Mar 18 10:20:52 thanos-store.kayrros thanos[31070]: level=warn ts=2020-03-18T10:20:52.085684529Z caller=streamed_block_writer.go:115 msg="empty chunks happened, skip series"
Mar 18 10:20:52 thanos-store.kayrros thanos[31070]: level=warn ts=2020-03-18T10:20:52.09119412Z caller=streamed_block_writer.go:115 msg="empty chunks happened, skip series" 
Mar 18 10:20:52 thanos-store.kayrros thanos[31070]: level=warn ts=2020-03-18T10:20:52.099343175Z caller=streamed_block_writer.go:115 msg="empty chunks happened, skip series"
Mar 18 10:20:52 thanos-store.kayrros thanos[31070]: level=warn ts=2020-03-18T10:20:52.10028563Z caller=streamed_block_writer.go:115 msg="empty chunks happened, skip series" 
Mar 18 10:20:52 thanos-store.kayrros thanos[31070]: level=warn ts=2020-03-18T10:20:52.106005072Z caller=streamed_block_writer.go:115 msg="empty chunks happened, skip series"
Mar 18 10:20:52 thanos-store.kayrros thanos[31070]: level=warn ts=2020-03-18T10:20:52.114047956Z caller=streamed_block_writer.go:115 msg="empty chunks happened, skip series"
Mar 18 10:20:52 thanos-store.kayrros thanos[31070]: level=warn ts=2020-03-18T10:20:52.115060124Z caller=streamed_block_writer.go:115 msg="empty chunks happened, skip series"
Mar 18 10:20:52 thanos-store.kayrros thanos[31070]: level=warn ts=2020-03-18T10:20:52.120673275Z caller=streamed_block_writer.go:115 msg="empty chunks happened, skip series"
Mar 18 10:20:52 thanos-store.kayrros thanos[31070]: level=warn ts=2020-03-18T10:20:52.128648416Z caller=streamed_block_writer.go:115 msg="empty chunks happened, skip series"
Mar 18 10:20:52 thanos-store.kayrros thanos[31070]: level=warn ts=2020-03-18T10:20:52.129532343Z caller=streamed_block_writer.go:115 msg="empty chunks happened, skip series"
Mar 18 10:20:52 thanos-store.kayrros thanos[31070]: level=warn ts=2020-03-18T10:20:52.135284875Z caller=streamed_block_writer.go:115 msg="empty chunks happened, skip series"
Mar 18 10:20:52 thanos-store.kayrros thanos[31070]: level=warn ts=2020-03-18T10:20:52.143215741Z caller=streamed_block_writer.go:115 msg="empty chunks happened, skip series"
Mar 18 10:20:52 thanos-store.kayrros thanos[31070]: level=warn ts=2020-03-18T10:20:52.144413163Z caller=streamed_block_writer.go:115 msg="empty chunks happened, skip series"
Mar 18 10:20:52 thanos-store.kayrros thanos[31070]: level=warn ts=2020-03-18T10:20:52.150230934Z caller=streamed_block_writer.go:115 msg="empty chunks happened, skip series"
Mar 18 10:20:52 thanos-store.kayrros thanos[31070]: level=warn ts=2020-03-18T10:20:52.162036188Z caller=streamed_block_writer.go:115 msg="empty chunks happened, skip series"
Mar 18 10:20:52 thanos-store.kayrros thanos[31070]: level=warn ts=2020-03-18T10:20:52.162904811Z caller=streamed_block_writer.go:115 msg="empty chunks happened, skip series"
Mar 18 10:20:52 thanos-store.kayrros thanos[31070]: level=warn ts=2020-03-18T10:20:52.169096613Z caller=streamed_block_writer.go:115 msg="empty chunks happened, skip series"
Mar 18 10:24:59 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T10:24:59.781066236Z caller=streamed_block_writer.go:185 msg="finalized downsampled block" mint=1
Mar 18 10:24:59 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T10:24:59.781147938Z caller=downsample.go:295 msg="downsampled block" from=01E3PGX1R5GA71BBDMM0MV
Mar 18 10:25:26 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T10:25:26.257086509Z caller=downsample.go:309 msg="uploaded block" id=01E3PJAMYS2PS7QPTW4Q1EA90V 
Mar 18 10:25:26 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T10:25:26.892881082Z caller=compact.go:319 msg="start second pass of downsampling"
Mar 18 10:25:27 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T10:25:27.235465806Z caller=fetcher.go:368 component=block.MetaFetcher msg="successfully fetched 
Mar 18 10:25:27 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T10:25:27.235889306Z caller=compact.go:324 msg="downsampling iterations done"
Mar 18 10:25:27 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T10:25:27.236032207Z caller=retention.go:20 msg="start optional retention"
Mar 18 10:25:27 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T10:25:27.509474806Z caller=fetcher.go:368 component=block.MetaFetcher msg="successfully fetched 
Mar 18 10:25:27 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T10:25:27.51000045Z caller=retention.go:41 msg="optional retention apply done"
Mar 18 10:25:27 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T10:25:27.510108489Z caller=clean.go:25 msg="started cleaning of aborted partial uploads"
Mar 18 10:25:27 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T10:25:27.897153216Z caller=fetcher.go:368 component=block.MetaFetcher msg="successfully fetched 
Mar 18 10:25:27 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T10:25:27.909853174Z caller=clean.go:50 msg="cleaning of aborted partial uploads done"
Mar 18 10:25:28 thanos-store.kayrros thanos[31070]: level=warn ts=2020-03-18T10:25:27.957940476Z caller=intrumentation.go:58 msg="changing probe status" status=not-ready rea
Mar 18 10:25:28 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T10:25:28.023761834Z caller=http.go:81 service=http/server component=compact msg="internal server
Mar 18 10:25:28 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T10:25:28.024002433Z caller=intrumentation.go:70 msg="changing probe status" status=not-healthy r
Mar 18 10:25:28 thanos-store.kayrros thanos[31070]: level=info ts=2020-03-18T10:25:28.061635194Z caller=main.go:213 msg=exiting

Logs from starting manually the service with debug log.level

$ thanos compact --data-dir=~/compact --objstore.config-file=/etc/thanos/bucket_config.yml --log.level=debug
level=debug ts=2020-03-18T13:35:52.282711035Z caller=main.go:104 msg="maxprocs: Leaving GOMAXPROCS=[4]: CPU quota undefined"
level=info ts=2020-03-18T13:35:52.282968048Z caller=main.go:152 msg="Tracing will be disabled"
level=info ts=2020-03-18T13:35:52.283285025Z caller=factory.go:46 msg="loading bucket configuration"
level=info ts=2020-03-18T13:35:52.283955858Z caller=compact.go:386 msg="starting compact node"
level=info ts=2020-03-18T13:35:52.283984913Z caller=intrumentation.go:52 msg="changing probe status" status=ready
level=info ts=2020-03-18T13:35:52.284049926Z caller=intrumentation.go:64 msg="changing probe status" status=healthy
level=info ts=2020-03-18T13:35:52.284064254Z caller=compact.go:858 msg="start sync of metas"
level=info ts=2020-03-18T13:35:52.284084813Z caller=http.go:56 service=http/server component=compact msg="listening for requests and metrics" address=0.0.0.0:10902
level=debug ts=2020-03-18T13:35:52.550408578Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3FRKYBYSTD45WEEJ1KV4XC2/meta.json
level=debug ts=2020-03-18T13:35:52.578419892Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PGMK5TBEGET8J49G325C0A/meta.json
level=debug ts=2020-03-18T13:35:52.579690156Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PVNC8M421ZKKC6GFHDSBQ3/meta.json
level=debug ts=2020-03-18T13:35:52.583354801Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PJAMYS2PS7QPTW4Q1EA90V/meta.json
level=debug ts=2020-03-18T13:35:52.598206364Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PETKSRER1Y0PRYVGQNWZ9S/meta.json
level=debug ts=2020-03-18T13:35:52.610872679Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PPKA5C2YA7Q9SV21VXWMY4/meta.json
level=debug ts=2020-03-18T13:35:52.61187577Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E39HYB94V39GYT9T6KA131K2/meta.json
level=debug ts=2020-03-18T13:35:52.615649594Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PVNC77FR5YDDMKQW464VG2/meta.json
level=debug ts=2020-03-18T13:35:52.615840852Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PGJK1JM5CB6ZN449HQCE0F/meta.json
level=debug ts=2020-03-18T13:35:52.61672472Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PGSA1PB7CZKS1G181Q72RV/meta.json
level=debug ts=2020-03-18T13:35:52.625459515Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PG2YSQ17D0KWCCXWAX4K6S/meta.json
level=debug ts=2020-03-18T13:35:52.625461713Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PFMJHAA0J71W52TWQKFMHE/meta.json
level=debug ts=2020-03-18T13:35:52.630724676Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PHPHW0E4WS0E1ZBBZT3H07/meta.json
level=debug ts=2020-03-18T13:35:52.637320312Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3ER1AGHW6XAQCB1QP7PDEDW/meta.json
level=debug ts=2020-03-18T13:35:52.63795826Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PG6Z7031SH3ZSQ3RM0MC33/meta.json
level=debug ts=2020-03-18T13:35:52.638698353Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PGC20QPPH2TB0TDMR156PQ/meta.json
level=debug ts=2020-03-18T13:35:52.64096445Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PED0BMA2FET2YBPXSKGN21/meta.json
level=debug ts=2020-03-18T13:35:52.641714558Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PHSPD6AW2Z7CKCDQF4W8KT/meta.json
level=debug ts=2020-03-18T13:35:52.641814883Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PGX1R5GA71BBDMM0MVHP2G/meta.json
level=debug ts=2020-03-18T13:35:52.653557994Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PG8KWQS3WGW0ZYMF54QM67/meta.json
level=debug ts=2020-03-18T13:35:52.656515599Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PPH4YAW46Q0KVEVVFVWJW1/meta.json
level=debug ts=2020-03-18T13:35:52.660450594Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PFXN6N506B49E6QBGQZTQS/meta.json
level=debug ts=2020-03-18T13:35:52.661580324Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PEMWW0S1K3JHX7V4KGXR6W/meta.json
level=debug ts=2020-03-18T13:35:52.669676029Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PEWVF1R7SHTJRRYCQ2ABDQ/meta.json
level=debug ts=2020-03-18T13:35:52.70878147Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PMSN0M1QWKZNX3CQ4YYDGB/meta.json
level=debug ts=2020-03-18T13:35:52.721290129Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PGQW8K3J46ZG7GSR836KT2/meta.json
level=debug ts=2020-03-18T13:35:52.729989637Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PMSMZ9NM4RR2RHV6EEXD7D/meta.json
level=debug ts=2020-03-18T13:35:52.789896288Z caller=fetcher.go:196 component=block.MetaFetcher msg="download meta" name=01E3PH9SNGAJZD4A8Z3V5QE6A8/meta.json
level=info ts=2020-03-18T13:35:52.988832664Z caller=fetcher.go:368 component=block.MetaFetcher msg="successfully fetched block metadata" duration=704.732928ms cached=28 returned=28 partial=0
level=info ts=2020-03-18T13:35:52.988884193Z caller=compact.go:864 msg="start of GC"
level=info ts=2020-03-18T13:35:52.988898715Z caller=compact.go:872 msg="start of compaction"
level=info ts=2020-03-18T13:35:53.02221433Z caller=compact.go:910 msg="compaction iterations done"
level=info ts=2020-03-18T13:35:53.022302628Z caller=compact.go:313 msg="start first pass of downsampling"
level=info ts=2020-03-18T13:35:53.378101437Z caller=fetcher.go:368 component=block.MetaFetcher msg="successfully fetched block metadata" duration=355.743697ms cached=28 returned=28 partial=0
level=info ts=2020-03-18T13:35:53.378366368Z caller=compact.go:319 msg="start second pass of downsampling"
level=info ts=2020-03-18T13:35:53.658777926Z caller=fetcher.go:368 component=block.MetaFetcher msg="successfully fetched block metadata" duration=280.272829ms cached=28 returned=28 partial=0
level=info ts=2020-03-18T13:35:53.658893741Z caller=compact.go:324 msg="downsampling iterations done"
level=info ts=2020-03-18T13:35:53.658909622Z caller=retention.go:20 msg="start optional retention"
level=info ts=2020-03-18T13:35:53.952949659Z caller=fetcher.go:368 component=block.MetaFetcher msg="successfully fetched block metadata" duration=294.026254ms cached=28 returned=28 partial=0
level=info ts=2020-03-18T13:35:53.953000469Z caller=retention.go:41 msg="optional retention apply done"
level=info ts=2020-03-18T13:35:53.95301556Z caller=clean.go:25 msg="started cleaning of aborted partial uploads"
level=info ts=2020-03-18T13:35:54.189909204Z caller=fetcher.go:368 component=block.MetaFetcher msg="successfully fetched block metadata" duration=236.882414ms cached=28 returned=28 partial=0
level=info ts=2020-03-18T13:35:54.189970605Z caller=clean.go:50 msg="cleaning of aborted partial uploads done"
level=warn ts=2020-03-18T13:35:54.189995457Z caller=intrumentation.go:58 msg="changing probe status" status=not-ready reason=null
level=info ts=2020-03-18T13:35:54.190042138Z caller=http.go:81 service=http/server component=compact msg="internal server shutdown" err=null
level=info ts=2020-03-18T13:35:54.190055398Z caller=intrumentation.go:70 msg="changing probe status" status=not-healthy reason=null
level=info ts=2020-03-18T13:35:54.190079534Z caller=main.go:213 msg=exiting
level=debug ts=2020-03-18T13:35:54.190095924Z caller=main.go:104 msg="maxprocs: No GOMAXPROCS change to reset%!(EXTRA []interface {}=[])"

Anything else we need to know:
I am not sure if that could be related to this and to understand the implications of running both services (compact and store) in the same environment

@Gnoale Gnoale changed the title compact: shutdown unexpectidly compact: shutdown unexpectedly Mar 18, 2020
@GiedriusS
Copy link
Member

It has done what you've asked it to do - it has done one iteration and exited :P you can make it work continuously by passing --wait. Let me know if that helps.

@Gnoale
Copy link
Author

Gnoale commented Mar 18, 2020

Oh ! all good then :D

@Gnoale Gnoale closed this as completed Mar 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants