compact: data corruption during downsapmle, test and fix. (#6598)

* Samples to reproduce data corruption during downsapmle, tests and fix. Signed-off-by: Vasiliy Rumyantsev <[email protected]> * Samples to reproduce data corruption during downsapmle, tests and fix. Signed-off-by: Vasiliy Rumyantsev <[email protected]> * added test for chunk with NaN values only Signed-off-by: Vasiliy Rumyantsev <[email protected]> * CHANGELOG.md Signed-off-by: Vasiliy Rumyantsev <[email protected]> * added check for math.NaN Signed-off-by: Vasiliy Rumyantsev <[email protected]> * optimized NaN checking Signed-off-by: Vasiliy Rumyantsev <[email protected]> --------- Signed-off-by: Vasiliy Rumyantsev <[email protected]>
thanos-io · Sep 12, 2023 · 2ec9b0c · 2ec9b0c
1 parent f6a3950
commit 2ec9b0c
Show file tree

Hide file tree

Showing 3 changed files with 710 additions and 1 deletion.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -109,6 +109,7 @@ We use *breaking :warning:* to mark changes that are not backward compatible (re
 - [#6592](https://github.com/thanos-io/thanos/pull/6592) Query Frontend: fix bugs in vertical sharding `without` and `union` function to allow more queries to be shardable.
 - [#6317](https://github.com/thanos-io/thanos/pull/6317) *: Fix internal label deduplication bug, by resorting store response set.
 - [#6189](https://github.com/thanos-io/thanos/pull/6189) Rule: Fix panic when calling API `/api/v1/rules?type=alert`.
+- [#6598](https://github.com/thanos-io/thanos/pull/6598) compact: fix data corruption with "invalid size" error during downsample
 
 ### Changed
 - [#6049](https://github.com/thanos-io/thanos/pull/6049) Compact: *breaking :warning:* Replace group with resolution in compact metrics to avoid cardinality explosion on compact metrics for large numbers of groups.

diff --git a/pkg/compact/downsample/downsample.go b/pkg/compact/downsample/downsample.go
@@ -372,8 +372,17 @@ func downsampleRawLoop(data []sample, resolution int64, numChunks int) []chunks.
 		for ; j < len(data) && data[j].t <= curW; j++ {
 		}
 
-		batch := data[:j]
+		batch := make([]sample, 0, j)
+		for _, s := range data[:j] {
+			if math.IsNaN(s.v) {
+				continue
+			}
+			batch = append(batch, s)
+		}
 		data = data[j:]
+		if len(batch) == 0 {
+			continue
+		}
 
 		ab := newAggrChunkBuilder()