[SPARK-49038][SQL] SQLMetric should report the raw value in the accumulator update event #47721
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
cc @viirya |
There was a problem hiding this comment.
To @cloud-fan , do you think we can have a new test case to verify and protect this behavior change in the future?
+1 to have a new test. |
|
@dongjoon-hyun This is really beyond my expertise... we have some simple UI tests for the HTML content, but the SQL plan UI is SVG AFAIK. @yaooqinn do you have any ideas? |
|
Got it. Thank you for the reply, @cloud-fan . |
|
@cloud-fan If this helps, we can try reusing the new test I added in #47516 with some minor tweaks to protect this behavior. Given UI see -1 value now after this change, it gets filtered out correctly by |
|
@virrrat The problem is we need to test that the input to |
|
Got it, makes sense. |
|
Maybe just add a test to verify |
Testing this change with the UI is a bit too heavy, I think we can test it directly in SQLMetricsSuite |
dongjoon-hyun
approved these changes
Aug 13, 2024
|
Merged to master. Thank you, @cloud-fan and all. There is a conflict on branch-3.5 from |
cloud-fan
added a commit
to cloud-fan/spark
that referenced
this pull request
Aug 14, 2024
…ulator update event Some `SQLMetrics` set the initial value to `-1`, so that we can recognize no-update metrics (e.g. there is no input data and the metric is not updated at all) and filter them out later in the UI. However, there is a bug here. Spark turns accumulator updates into `AccumulableInfo`, using `AccumulatorV2#value`. To avoid exposing the internal `-1` value to end users, `SQLMetric#value` turns `-1` into `0` before returning the value. See more details in apache#39311 . UI can no longer see `-1` and filter them out. This PR fixes the bug by using the raw value of `SQLMetric` to create `AccumulableInfo`, so that UI can still see `-1` and filters it. To avoid getting the wrong min value for certain SQL metrics when some partitions have no data. Yes, if people write spark listeners to watch the `SparkListenerExecutorMetricsUpdate` event, they can see the correct value of SQL metrics. manual UI tests. We do not have an end-to-end UI test framework for SQL metrics yet. no Closes apache#47721 from cloud-fan/metrics. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
dongjoon-hyun
pushed a commit
that referenced
this pull request
Aug 14, 2024
…accumulator update event backport #47721 to 3.5 ### What changes were proposed in this pull request? Some `SQLMetrics` set the initial value to `-1`, so that we can recognize no-update metrics (e.g. there is no input data and the metric is not updated at all) and filter them out later in the UI. However, there is a bug here. Spark turns accumulator updates into `AccumulableInfo`, using `AccumulatorV2#value`. To avoid exposing the internal `-1` value to end users, `SQLMetric#value` turns `-1` into `0` before returning the value. See more details in #39311 . UI can no longer see `-1` and filter them out. This PR fixes the bug by using the raw value of `SQLMetric` to create `AccumulableInfo`, so that UI can still see `-1` and filters it. ### Why are the changes needed? To avoid getting the wrong min value for certain SQL metrics when some partitions have no data. ### Does this PR introduce _any_ user-facing change? Yes, if people write spark listeners to watch the `SparkListenerExecutorMetricsUpdate` event, they can see the correct value of SQL metrics. ### How was this patch tested? manual UI tests. We do not have an end-to-end UI test framework for SQL metrics yet. ### Was this patch authored or co-authored using generative AI tooling? no Closes #47749 from cloud-fan/branch-3.5. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
dongjoon-hyun
pushed a commit
that referenced
this pull request
Aug 14, 2024
…accumulator update event backport #47721 to 3.5 Some `SQLMetrics` set the initial value to `-1`, so that we can recognize no-update metrics (e.g. there is no input data and the metric is not updated at all) and filter them out later in the UI. However, there is a bug here. Spark turns accumulator updates into `AccumulableInfo`, using `AccumulatorV2#value`. To avoid exposing the internal `-1` value to end users, `SQLMetric#value` turns `-1` into `0` before returning the value. See more details in #39311 . UI can no longer see `-1` and filter them out. This PR fixes the bug by using the raw value of `SQLMetric` to create `AccumulableInfo`, so that UI can still see `-1` and filters it. To avoid getting the wrong min value for certain SQL metrics when some partitions have no data. Yes, if people write spark listeners to watch the `SparkListenerExecutorMetricsUpdate` event, they can see the correct value of SQL metrics. manual UI tests. We do not have an end-to-end UI test framework for SQL metrics yet. no Closes #47749 from cloud-fan/branch-3.5. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit bd2cbd6) Signed-off-by: Dongjoon Hyun <[email protected]>
IvanK-db
pushed a commit
to IvanK-db/spark
that referenced
this pull request
Sep 20, 2024
…ulator update event ### What changes were proposed in this pull request? Some `SQLMetrics` set the initial value to `-1`, so that we can recognize no-update metrics (e.g. there is no input data and the metric is not updated at all) and filter them out later in the UI. However, there is a bug here. Spark turns accumulator updates into `AccumulableInfo`, using `AccumulatorV2#value`. To avoid exposing the internal `-1` value to end users, `SQLMetric#value` turns `-1` into `0` before returning the value. See more details in apache#39311 . UI can no longer see `-1` and filter them out. This PR fixes the bug by using the raw value of `SQLMetric` to create `AccumulableInfo`, so that UI can still see `-1` and filters it. ### Why are the changes needed? To avoid getting the wrong min value for certain SQL metrics when some partitions have no data. ### Does this PR introduce _any_ user-facing change? Yes, if people write spark listeners to watch the `SparkListenerExecutorMetricsUpdate` event, they can see the correct value of SQL metrics. ### How was this patch tested? manual UI tests. We do not have an end-to-end UI test framework for SQL metrics yet. ### Was this patch authored or co-authored using generative AI tooling? no Closes apache#47721 from cloud-fan/metrics. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
szehon-ho
pushed a commit
to szehon-ho/spark
that referenced
this pull request
Sep 24, 2024
…accumulator update event backport apache#47721 to 3.5 Some `SQLMetrics` set the initial value to `-1`, so that we can recognize no-update metrics (e.g. there is no input data and the metric is not updated at all) and filter them out later in the UI. However, there is a bug here. Spark turns accumulator updates into `AccumulableInfo`, using `AccumulatorV2#value`. To avoid exposing the internal `-1` value to end users, `SQLMetric#value` turns `-1` into `0` before returning the value. See more details in apache#39311 . UI can no longer see `-1` and filter them out. This PR fixes the bug by using the raw value of `SQLMetric` to create `AccumulableInfo`, so that UI can still see `-1` and filters it. To avoid getting the wrong min value for certain SQL metrics when some partitions have no data. Yes, if people write spark listeners to watch the `SparkListenerExecutorMetricsUpdate` event, they can see the correct value of SQL metrics. manual UI tests. We do not have an end-to-end UI test framework for SQL metrics yet. no Closes apache#47749 from cloud-fan/branch-3.5. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit bd2cbd6) Signed-off-by: Dongjoon Hyun <[email protected]>
attilapiros
pushed a commit
to attilapiros/spark
that referenced
this pull request
Oct 4, 2024
…ulator update event ### What changes were proposed in this pull request? Some `SQLMetrics` set the initial value to `-1`, so that we can recognize no-update metrics (e.g. there is no input data and the metric is not updated at all) and filter them out later in the UI. However, there is a bug here. Spark turns accumulator updates into `AccumulableInfo`, using `AccumulatorV2#value`. To avoid exposing the internal `-1` value to end users, `SQLMetric#value` turns `-1` into `0` before returning the value. See more details in apache#39311 . UI can no longer see `-1` and filter them out. This PR fixes the bug by using the raw value of `SQLMetric` to create `AccumulableInfo`, so that UI can still see `-1` and filters it. ### Why are the changes needed? To avoid getting the wrong min value for certain SQL metrics when some partitions have no data. ### Does this PR introduce _any_ user-facing change? Yes, if people write spark listeners to watch the `SparkListenerExecutorMetricsUpdate` event, they can see the correct value of SQL metrics. ### How was this patch tested? manual UI tests. We do not have an end-to-end UI test framework for SQL metrics yet. ### Was this patch authored or co-authored using generative AI tooling? no Closes apache#47721 from cloud-fan/metrics. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
himadripal
pushed a commit
to himadripal/spark
that referenced
this pull request
Oct 19, 2024
…ulator update event ### What changes were proposed in this pull request? Some `SQLMetrics` set the initial value to `-1`, so that we can recognize no-update metrics (e.g. there is no input data and the metric is not updated at all) and filter them out later in the UI. However, there is a bug here. Spark turns accumulator updates into `AccumulableInfo`, using `AccumulatorV2#value`. To avoid exposing the internal `-1` value to end users, `SQLMetric#value` turns `-1` into `0` before returning the value. See more details in apache#39311 . UI can no longer see `-1` and filter them out. This PR fixes the bug by using the raw value of `SQLMetric` to create `AccumulableInfo`, so that UI can still see `-1` and filters it. ### Why are the changes needed? To avoid getting the wrong min value for certain SQL metrics when some partitions have no data. ### Does this PR introduce _any_ user-facing change? Yes, if people write spark listeners to watch the `SparkListenerExecutorMetricsUpdate` event, they can see the correct value of SQL metrics. ### How was this patch tested? manual UI tests. We do not have an end-to-end UI test framework for SQL metrics yet. ### Was this patch authored or co-authored using generative AI tooling? no Closes apache#47721 from cloud-fan/metrics. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Some
SQLMetricsset the initial value to-1, so that we can recognize no-update metrics (e.g. there is no input data and the metric is not updated at all) and filter them out later in the UI.However, there is a bug here. Spark turns accumulator updates into
AccumulableInfo, usingAccumulatorV2#value. To avoid exposing the internal-1value to end users,SQLMetric#valueturns-1into0before returning the value. See more details in #39311 . UI can no longer see-1and filter them out.This PR fixes the bug by using the raw value of
SQLMetricto createAccumulableInfo, so that UI can still see-1and filters it.Why are the changes needed?
To avoid getting the wrong min value for certain SQL metrics when some partitions have no data.
Does this PR introduce any user-facing change?
Yes, if people write spark listeners to watch the
SparkListenerExecutorMetricsUpdateevent, they can see the correct value of SQL metrics.How was this patch tested?
manual UI tests. We do not have an end-to-end UI test framework for SQL metrics yet.
Was this patch authored or co-authored using generative AI tooling?
no