[SPARK-23838][WEBUI] Running SQL query is displayed as "completed" in SQL tab #20955
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #88790 has finished for PR 20955 at commit
|
|
@vanzin Please help review this. |
gengliangwang
force-pushed
the
jobCompleted
branch
from
b418feb to
2357f59
Compare
|
Test build #88795 has finished for PR 20955 at commit
|
vanzin
reviewed
Apr 2, 2018
| @@ -39,7 +39,7 @@ private[ui] class AllExecutionsPage(parent: SQLTab) extends WebUIPage("") with L | |||
| val failed = new mutable.ArrayBuffer[SQLExecutionUIData]() | |||
|
|
|||
| sqlStore.executionsList().foreach { e => | |||
| val isRunning = e.jobs.exists { case (_, status) => status == JobExecutionStatus.RUNNING } | |||
| val isRunning = e.completionTime.isEmpty | |||
There was a problem hiding this comment.
I think you actually need both checks. The code in 2.2 is this:
if (executionUIData.completionTime.nonEmpty && !executionUIData.hasRunningJobs) {
// We are the last job of this execution, so mark the execution as finished. Note that
// `onExecutionEnd` also does this, but currently that can be called before `onJobEnd`
// since these are called on different threads.
markExecutionFinished(executionId)
}
The original reason why this code is like this is that job events and sql execution events could arrive out of order; I don't know if that is still true, but I tried to maintain the same workarounds in the new code.
If the out-of-order issue exists, then your change would introduce the opposite problem: an execution marked as completed when existing known jobs are still running, because the execution end event arrived before the job end event.
There was a problem hiding this comment.
Make sense. I have updated the code.
gengliangwang
force-pushed
the
jobCompleted
branch
from
2357f59 to
ceb28eb
Compare
|
Test build #88841 has finished for PR 20955 at commit
|
|
retest this please |
|
Test build #88857 has finished for PR 20955 at commit
|
|
Merging to master / 2.3. |
asfgit
pushed a commit
that referenced
this pull request
Apr 4, 2018
… SQL tab ## What changes were proposed in this pull request? A running SQL query would appear as completed in the Spark UI:  We can see the query in "Completed queries", while in in the job page we see it's still running Job 132.  After some time in the query still appears in "Completed queries" (while it's still running), but the "Duration" gets increased.  To reproduce, we can run a query with multiple jobs. E.g. Run TPCDS q6. The reason is that updates from executions are written into kvstore periodically, and the job start event may be missed. ## How was this patch tested? Manually run the job again and check the SQL Tab. The fix is pretty simple. Author: Gengliang Wang <[email protected]> Closes #20955 from gengliangwang/jobCompleted. (cherry picked from commit d8379e5) Signed-off-by: Marcelo Vanzin <[email protected]>
mshtelma
pushed a commit
to mshtelma/spark
that referenced
this pull request
Apr 5, 2018
… SQL tab ## What changes were proposed in this pull request? A running SQL query would appear as completed in the Spark UI:  We can see the query in "Completed queries", while in in the job page we see it's still running Job 132.  After some time in the query still appears in "Completed queries" (while it's still running), but the "Duration" gets increased.  To reproduce, we can run a query with multiple jobs. E.g. Run TPCDS q6. The reason is that updates from executions are written into kvstore periodically, and the job start event may be missed. ## How was this patch tested? Manually run the job again and check the SQL Tab. The fix is pretty simple. Author: Gengliang Wang <[email protected]> Closes apache#20955 from gengliangwang/jobCompleted.
robert3005
pushed a commit
to palantir/spark
that referenced
this pull request
Apr 7, 2018
… SQL tab ## What changes were proposed in this pull request? A running SQL query would appear as completed in the Spark UI:  We can see the query in "Completed queries", while in in the job page we see it's still running Job 132.  After some time in the query still appears in "Completed queries" (while it's still running), but the "Duration" gets increased.  To reproduce, we can run a query with multiple jobs. E.g. Run TPCDS q6. The reason is that updates from executions are written into kvstore periodically, and the job start event may be missed. ## How was this patch tested? Manually run the job again and check the SQL Tab. The fix is pretty simple. Author: Gengliang Wang <[email protected]> Closes apache#20955 from gengliangwang/jobCompleted.
peter-toth
pushed a commit
to peter-toth/spark
that referenced
this pull request
Oct 6, 2018
… SQL tab A running SQL query would appear as completed in the Spark UI:  We can see the query in "Completed queries", while in in the job page we see it's still running Job 132.  After some time in the query still appears in "Completed queries" (while it's still running), but the "Duration" gets increased.  To reproduce, we can run a query with multiple jobs. E.g. Run TPCDS q6. The reason is that updates from executions are written into kvstore periodically, and the job start event may be missed. Manually run the job again and check the SQL Tab. The fix is pretty simple. Author: Gengliang Wang <[email protected]> Closes apache#20955 from gengliangwang/jobCompleted. (cherry picked from commit d8379e5) Signed-off-by: Marcelo Vanzin <[email protected]> Change-Id: I779e8d5363150afe3045c774f77f6395972192f8
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
A running SQL query would appear as completed in the Spark UI:
We can see the query in "Completed queries", while in in the job page we see it's still running Job 132.
After some time in the query still appears in "Completed queries" (while it's still running), but the "Duration" gets increased.
To reproduce, we can run a query with multiple jobs. E.g. Run TPCDS q6.
The reason is that updates from executions are written into kvstore periodically, and the job start event may be missed.
How was this patch tested?
Manually run the job again and check the SQL Tab. The fix is pretty simple.