[SPARK-4168][WebUI] web statges number should show correctly when stages are more than 1000 #3035
Conversation
…ges are more than 1000
|
Test build #22608 has started for PR 3035 at commit
|
|
Test build #22608 has finished for PR 3035 at commit
|
|
Test PASSed. |
| // Number of completed stages, not equal to completedStages.size | ||
| var completedStagesNum = 0 | ||
| // number of failed stages, not equal to failedStagesNum.size | ||
| var failedStagesNum = 0 |
There was a problem hiding this comment.
I prefer to call these numFailedStages
|
Test build #22679 has started for PR 3035 at commit
|
|
Test build #22679 has finished for PR 3035 at commit
|
|
Test PASSed. |
|
@andrewor14 , sorry to misunderstand. the comments updated, can you have a look to see if it is ok? |
|
Test build #22717 has started for PR 3035 at commit
|
|
Test build #22717 has finished for PR 3035 at commit
|
|
Test PASSed. |
|
@andrewor14 , can you help to check whether this patch is fine now? |
|
Yeah LGTM I'll merge this. |
…ges are more than 1000 The number of completed stages and failed stages showed on webUI will always be less than 1000. This is really misleading when there are already thousands of stages completed or failed. The number should be correct even when only partial stages listed on the webUI (stage info will be removed if the number is too large). Author: Zhang, Liye <[email protected]> Closes #3035 from liyezhang556520/webStageNum and squashes the following commits: d9e29fb [Zhang, Liye] add detailed comments for variables 4ea8fd1 [Zhang, Liye] change variable name accroding to comments f4c404d [Zhang, Liye] [SPARK-4168][WebUI] web statges number should show correctly when stages are more than 1000
The number of completed stages and failed stages showed on webUI will always be less than 1000. This is really misleading when there are already thousands of stages completed or failed. The number should be correct even when only partial stages listed on the webUI (stage info will be removed if the number is too large).