MQE: Add NH support to binary operations (#8838)

* MQE: Complete binary operations For one-to-one mapping. * Add note to followup on optimisation * Fix moving the index across the slices * Remove comparison operators for now * Add InstantVectorSeriesDataIterator * Cleanup computeResult Use new InstantVectorSeriesDataIterator And properly assign/re-use slices * Resolve initial review feedback * Merge floats and histograms independantly before checking for conflicts * rename prepare slice functions * Keep track of indexes when filtering data * Remove filteredData struct and sort nil floats/histograms to list ends to track * Sort seriesIndices at the same time * add note * re-add unsupported operation * Fix time format * Add tests for histogram series merging * Fix lint * Rename some variables for clarity * Return all unused float slices * Check capacity not length when preparing slices * Move sourceSeriesMetadata out of seriesForOneGroupSide * Also fix histogram slice returning * Split out sorters for floats/histograms * Fix returning the correct slice type * Fix bug with pool usage * Expand testing to make sure histograms are as expected * Add tests (currently broken) * Fix re-use of slices with mixed points * Output first match group with conflicting points * Update changelog * Fix comment * Make sure expected histogram is returend in data_test * Add NH binop benchmark * Also benchmark NH + NH * Fix+extend comments * Don't export left/right iterator * Use better test naming * Extend NH binop tests * Tweak InstantVectorSeriesData readability * Add extra data test * Run merging tests with histograms
grafana · Aug 6, 2024 · 866dfc8 · 866dfc8
1 parent 124bc4e
commit 866dfc8
Show file tree

Hide file tree

Showing 7 changed files with 1,378 additions and 74 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -22,7 +22,7 @@
 * [CHANGE] Ingester client: experimental support for client-side circuit breakers, their configuration options (`-ingester.client.circuit-breaker.*`) and metrics (`cortex_ingester_client_circuit_breaker_results_total`, `cortex_ingester_client_circuit_breaker_transitions_total`) were removed. #8802
 * [CHANGE] Ingester: circuit breakers do not open in case of per-instance limit errors anymore. Opening can be triggered only in case of push and pull requests exceeding the configured duration. #8854
 * [CHANGE] Query-frontend: Return `413 Request Entity Too Large` if a response shard for an `/active_series` request is too large. #8861
-* [FEATURE] Querier: add experimental streaming PromQL engine, enabled with `-querier.query-engine=mimir`. #8422 #8430 #8454 #8455 #8360 #8490 #8508 #8577 #8660 #8671 #8677 #8747 #8850 #8872
+* [FEATURE] Querier: add experimental streaming PromQL engine, enabled with `-querier.query-engine=mimir`. #8422 #8430 #8454 #8455 #8360 #8490 #8508 #8577 #8660 #8671 #8677 #8747 #8850 #8872 #8838
 * [FEATURE] Experimental Kafka-based ingest storage. #6888 #6894 #6929 #6940 #6951 #6974 #6982 #7029 #7030 #7091 #7142 #7147 #7148 #7153 #7160 #7193 #7349 #7376 #7388 #7391 #7393 #7394 #7402 #7404 #7423 #7424 #7437 #7486 #7503 #7508 #7540 #7621 #7682 #7685 #7694 #7695 #7696 #7697 #7701 #7733 #7734 #7741 #7752 #7838 #7851 #7871 #7877 #7880 #7882 #7887 #7891 #7925 #7955 #7967 #8031 #8063 #8077 #8088 #8135 #8176 #8184 #8194 #8216 #8217 #8222 #8233 #8503 #8542 #8579 #8657 #8686 #8688 #8703 #8706 #8708 #8738 #8750 #8778 #8808 #8809 #8841 #8842 #8845 #8853
   * What it is:
     * When the new ingest storage architecture is enabled, distributors write incoming write requests to a Kafka-compatible backend, and the ingesters asynchronously replay ingested data from Kafka. In this architecture, the write and read path are de-coupled through a Kafka-compatible backend. The write path and Kafka load is a function of the incoming write traffic, the read path load is a function of received queries. Whatever the load on the read path, it doesn't affect the write path.

diff --git a/pkg/streamingpromql/benchmarks/benchmarks.go b/pkg/streamingpromql/benchmarks/benchmarks.go
@@ -130,6 +130,12 @@ func TestCases(metricSizes []int) []BenchCase {
 			Expr:  "a_X - b_X",
 			Steps: 10000,
 		},
+		{
+			Expr: "nh_X + nh_X",
+		},
+		{
+			Expr: "nh_X / a_X",
+		},
 		// Test the case where one side of a binary operation has many more series than the other.
 		{
 			Expr: `a_100{l=~"[13579]."} - b_100`,