Improves the size estimation when cutting new rows (#3038)

* Improves the size estimation when cutting new rows Signed-off-by: Joe Elliott <[email protected]> * changelog Signed-off-by: Joe Elliott <[email protected]> * lint Signed-off-by: Joe Elliott <[email protected]> * changelog fix Signed-off-by: Joe Elliott <[email protected]> * adjust magic factor to be slightly more aggressive Signed-off-by: Joe Elliott <[email protected]> * wrong vresion Signed-off-by: Joe Elliott <[email protected]> --------- Signed-off-by: Joe Elliott <[email protected]>
grafana · Oct 20, 2023 · 0542bc3 · 0542bc3
1 parent b7f2956
commit 0542bc3
Show file tree

Hide file tree

Showing 2 changed files with 21 additions and 1 deletion.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -18,6 +18,7 @@
 * [CHANGE] Restructure Azure backends into versioned backends.  Introduce `use_v2_sdk` config option for switching. [#2952](https://github.com/grafana/tempo/issues/2952) (@zalegrala)
     v1: [azure-storage-blob-go](https://github.com/Azure/azure-storage-blob-go) original (now deprecated) SDK
     v2: [azure-sdk-for-go](https://github.com/Azure/azure-sdk-for-go)
+* [CHANGE] Adjust trace size estimation to better honor row group size settings. [#3038](https://github.com/grafana/tempo/pull/3038) (@joe-elliott)
 * [CHANGE] Update alpine image version to 3.18. [#3046](https://github.com/grafana/tempo/pull/) (@joe-elliott)
 * [ENHANCEMENT] Add block indexes to vParquet2 and vParquet3 to improve trace by ID lookup [#2697](https://github.com/grafana/tempo/pull/2697) (@mdisibio)
 * [ENHANCEMENT] Assert ingestion rate limits as early as possible [#2640](https://github.com/grafana/tempo/pull/2703) (@mghildiy)

diff --git a/tempodb/encoding/vparquet3/compactor.go b/tempodb/encoding/vparquet3/compactor.go
@@ -315,7 +315,26 @@ func estimateProtoSizeFromParquetRow(row parquet.Row) (size int) {
 // estimateMarshalledSizeFromParquetRow estimates the byte size as marshalled into parquet.
 // this is a very rough estimate and is generally 66%-100% of actual size.
 func estimateMarshalledSizeFromParquetRow(row parquet.Row) (size int) {
-	return len(row)
+	for _, v := range row {
+		sz := 1
+
+		switch v.Kind() {
+		case parquet.ByteArray:
+			sz = len(v.ByteArray())
+
+		case parquet.FixedLenByteArray:
+			sz = len(v.ByteArray())
+		}
+
+		if sz > 1 {
+			// for larger values, estimate 1 byte per 20 bytes. this is a very rough estimate
+			// but works well enough for our purposes. TODO: improve this calculation
+			sz = max(sz/20, 1) // nolint: typecheck
+		}
+
+		size += sz
+	}
+	return
 }
 
 // countSpans counts the number of spans in the given trace in deconstructed