cortex panic in minio when handle minio errors.

[t00350320]

Cortex panic when handle minio errors.
log message:
panic: send on closed channel

goroutine 799692642 [running]:
github.com/minio/minio-go/v7.(*Object).doGetRequest(_, {{0x0, 0x0, 0x0}, 0x0, 0x0, 0x0, 0x0, 0x1, 0x1, ...})
/home/opensource/cortex/vendor/github.com/minio/minio-go/v7/api-get-object.go:319 +0xdc
github.com/minio/minio-go/v7.(*Object).Read(0xc1586fc8c0, {0x0?, 0x0?, 0x0?})
/home/opensource/cortex/vendor/github.com/minio/minio-go/v7/api-get-object.go:399 +0x248
github.com/thanos-io/objstore/providers/s3.(*Bucket).getRange(0xc0005c7ea0, {0x26a80f8, 0xc79c618280}, {0xc01a331350, 0x2d}, 0xe26950, 0x862c)
/home/opensource/cortex/vendor/github.com/thanos-io/objstore/providers/s3/s3.go:428 +0x20e
github.com/thanos-io/objstore/providers/s3.(*Bucket).GetRange(0x7f519b3112f8?, {0x26a80f8?, 0xc79c618280?}, {0xc01a331350?, 0x0?}, 0x456137?, 0xc15616a998?)
/home/opensource/cortex/vendor/github.com/thanos-io/objstore/providers/s3/s3.go:445 +0x28
github.com/thanos-io/objstore.(*metricBucket).GetRange(0xc000469700, {0x26a80f8, 0xc79c618280}, {0xc01a331350, 0x2d}, 0x0?, 0x0?)
/home/opensource/cortex/vendor/github.com/thanos-io/objstore/objstore.go:526 +0xbd
github.com/thanos-io/objstore.TracingBucket.GetRange({{0x26bc070?, 0xc000469700?}}, {0x26a80f8?, 0xc79c618280?}, {0xc01a331350, 0x2d}, 0xc1f34ddc20?, 0x28?)
/home/opensource/cortex/vendor/github.com/thanos-io/objstore/tracing.go:50 +0x22c
github.com/cortexproject/cortex/pkg/storage/bucket.(*PrefixedBucketClient).GetRange(0xc041f1ffe0, {0x26a80f8, 0xc79c618280}, {0xc1f34ddc20?, 0x28}, 0x1de3700?, 0x36f79b0?)
/home/opensource/cortex/pkg/storage/bucket/prefixed_bucket_client.go:63 +0x97
github.com/cortexproject/cortex/pkg/storage/bucket.(*SSEBucketClient).GetRange(0xc1efd217a0?, {0x26a80f8?, 0xc79c618280?}, {0xc1f34ddc20?, 0x28?}, 0xe26950?, 0x862c?)
/home/opensource/cortex/pkg/storage/bucket/sse_bucket_client.go:109 +0x31
github.com/thanos-io/thanos/pkg/store.(*bucketBlock).chunkRangeReader(0xc13fa33c00?, {0x26a80f8?, 0xc79c618280?}, 0x7f519b313878?, 0xc0b886a3f8?, 0x40f5c7?)
/home/opensource/cortex/vendor/github.com/thanos-io/thanos/pkg/store/bucket.go:1769 +0xb0
github.com/thanos-io/thanos/pkg/store.(*bucketChunkReader).loadChunks(0xc159021310, {0x26a80f8, 0xc79c618280}, {0xca9e6e8000, 0x4, 0x0?}, {0x0, 0x0, 0x0}, 0x0, ...)
/home/opensource/cortex/vendor/github.com/thanos-io/thanos/pkg/store/bucket.go:2551 +0xd3

To Reproduce
Steps to reproduce the behavior:

1. Start Cortex (SHA or version)
   last commit 01dc235
   Author: Alan Protasio [email protected]
   Date: Wed Sep 21 18:19:11 2022 -0700

Expected behavior
no panic

Environment:

• Infrastructure: [e.g., Kubernetes]
• Deployment tool: [e.g.,binary]

[alanprot]

This seems related to thanos-io/thanos#5701 / thanos-io/thanos#5702

This should be fixed on the latest commit.

[songjiayang]

Maybe we should do another test with latest commit.

[yeya24]

Let's close this if you test everything is ok.
I believe this was fixed already with that pr in Thanos.

[t00350320]

ok.

[t00350320]

everything is ok in last week.