[SPARK-12902] [SQL] visualization for generated operators

[davies]

This PR brings back visualization for generated operators, they looks like:

Note: SQL metrics are not supported right now, because they are very slow, will be supported once we have batch mode.

[davies]

cc @zsxwing

[rxin]

This is pretty cool.

[SparkQA]

Test build #49670 has finished for PR 10828 at commit f0f3da6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #2412 has finished for PR 10828 at commit 331b46d.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zsxwing]

@davies did you check if the history server displays this correctly?

[zsxwing]

@davies did you check if the history server displays this correctly?

Just tried it in the history server and it worked correctly. Nice!

[zsxwing]

Why use a different style? I think we should make the UI stuff consistent.

[davies]

Which one will you suggest? This is outer of operator, should be lighter than that, I choose this because it's between white and the one used for operators.

[zsxwing]

I meant using the following style:

#plan-viz-graph svg g.cluster rect {
  fill: #A0DFFF;
  stroke: #3EC0FF;
  stroke-width: 1px;
}

This looks more consistent with the stage page:

By the way, I saw a bug in the stage page:

Could you check it?

[zsxwing]

LGTM except the inconsistent style.

[davies]

@zsxwing Updated, check that again.

[zsxwing]

LGTM pending tests

[SparkQA]

Test build #2421 has finished for PR 10828 at commit a179e8d.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zsxwing]

retest this please

[SparkQA]

Test build #49803 has finished for PR 10828 at commit ccac152.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #2423 has finished for PR 10828 at commit ccac152.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zsxwing]

retest this please

[SparkQA]

Test build #2424 has finished for PR 10828 at commit ccac152.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #2425 has finished for PR 10828 at commit ccac152.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #49825 has finished for PR 10828 at commit ccac152.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #49812 has finished for PR 10828 at commit ccac152.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #2426 has finished for PR 10828 at commit f5c9087.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #2427 has finished for PR 10828 at commit f5c9087.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #2428 has finished for PR 10828 at commit f5c9087.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #2430 has finished for PR 10828 at commit 163a014.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #2434 has finished for PR 10828 at commit 163a014.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #2435 has finished for PR 10828 at commit 163a014.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #2446 has finished for PR 10828 at commit 163a014.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #2455 has finished for PR 10828 at commit 163a014.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

It passed finally, bingo! Merging this into master, thanks!