[SPARK-5100][SQL] add thriftserver-ui support #3946

tianyi · 2015-01-08T08:54:56Z

In the latest Spark release, there is a Spark Streaming tab on the driver web UI, which shows information about running streaming application. It should be helpful for providing a monitor page in Thrift server because both streaming and Thrift server are long-term applications, and the details of the application do not show on stage page or job page.

Design doc is here: https://issues.apache.org/jira/secure/attachment/12690744/Spark%20Thrift-server%20monitor%20page.pdf

Prototype snapshot: https://issues.apache.org/jira/secure/attachment/12690297/prototype-screenshot.png

liancheng · 2015-01-08T08:56:20Z

ok to test

liancheng · 2015-01-08T08:58:15Z

...e-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala

@@ -93,6 +94,13 @@ private[hive] class HiveThriftServer2(hiveContext: HiveContext)
  extends HiveServer2
  with ReflectedCompositeService {

+  private[hive] val uiTab: Option[ThriftServerTab] =
+    if (hiveContext.hiveconf.getBoolean("spark.ui.enabled", true)) {


It's a bit weird to read Spark configuration via HiveConf although it's valid here. hiveContext.sparkContext.getConf might be better.

liancheng · 2015-01-08T09:57:55Z

ok to test

liancheng · 2015-01-08T09:58:46Z

...e-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala

@@ -28,6 +28,7 @@ import org.apache.spark.annotation.DeveloperApi
 import org.apache.spark.sql.hive.HiveContext
 import org.apache.spark.sql.hive.thriftserver.ReflectionUtils._
 import org.apache.spark.scheduler.{SparkListenerApplicationEnd, SparkListener}
+import org.apache.spark.sql.hive.thriftserver.ui.ThriftServerTab


Seems that imports are not ordered properly in this file...

SparkQA · 2015-01-08T10:03:24Z

Test build #25219 has finished for PR 3946 at commit f796a81.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait ThriftServerEventListener
- class SessionInfo(val session: HiveSession, val startTimestamp: Long)
- class ExecutionInfo(val statement: String, val session: HiveSession, val startTimestamp: Long)
- class ThriftServerUIEventListener(val conf: SparkConf)

liancheng · 2015-01-08T10:06:35Z

...hive-thriftserver/v0.12.0/src/main/scala/org/apache/spark/sql/hive/thriftserver/Shim12.scala

@@ -184,8 +184,10 @@ private[hive] class SparkExecuteStatementOperation(
  }

  def run(): Unit = {
+    val sid = UUID.randomUUID().toString


I guess we can use parentSession.getSessionHandle.getSessionId instead?

No, the listener needs a statement id, not a session id, for updating each statement's state.

liancheng · 2015-01-08T10:20:15Z

@tianyi I understand that because Hive 0.13.1 made incompatible changes to SessionManager.openSession, and you'd like to avoid adding new stuff to the shim layer. However, once a shim is introduced, it's doomed to grow :-) I think we can have a SparkSQLSessionManagerBase in the shim layer, which only handles openSession, and then make SparkSQLSessionManager extend it.

@scwf Would you mind to take a look at this? Particularly the job group related part.

scwf · 2015-01-08T10:33:30Z

...thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala

+
+import scala.xml.Node
+
+/** Page for Spark Web UI that shows statistics of a streaming job */


streaming job?

tianyi · 2015-01-16T07:39:47Z

I mean I can't match a job and a execution via statement, if there are two same sql running

On Jan 16, 2015, at 11:21, Fei Wang [email protected] wrote:

but it would get wrong when the thrift server is executing two same SQL at the same time.

you mean using two beeline to execute sql(must the same sql?) at same time will get error? what error you get?

—
Reply to this email directly or view it on GitHub.

tianyi · 2015-01-19T04:54:10Z

rebased from latest master.

SparkQA · 2015-01-19T05:01:33Z

Test build #25735 has finished for PR 3946 at commit daed3d1.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-01-19T05:56:02Z

Test build #25733 has finished for PR 3946 at commit 14a461d.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2015-01-19T06:21:08Z

Test build #25736 has finished for PR 3946 at commit fb507df.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

liancheng · 2015-01-19T09:13:36Z

...iftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLSessionManager.scala

  with ReflectedCompositeService {

  private lazy val sparkSqlOperationManager = new SparkSQLOperationManager(hiveContext)

  override def init(hiveConf: HiveConf) {
-    setSuperField(this, "hiveConf", hiveConf)
+


Remove this empty line.

use statementId as groupId to support multiple jobs for one sql. support multiple jobs for one sql in ThriftServerPage.

SparkQA · 2015-01-20T05:50:03Z

Test build #25792 has finished for PR 3946 at commit 32332cb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

scwf · 2015-01-21T04:18:51Z

part of group id is look ok to me. Now we create a uuid for each statement, and save it to EventListener, later we can make a follow up PR to add the logical that when user presses ctrl-c, use the group id of the statement to cancel the job. Maybe we need add a onStatementStop interface to EventListener for that.

liancheng · 2015-01-21T05:22:48Z

Actually the CTRL-C part resides in Beeline. Beeline does implemented a signal handler for this, but the registration code is commented out, don't know why...

…tServer2

SparkQA · 2015-01-26T08:40:22Z

Test build #26085 has finished for PR 3946 at commit fd35262.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

marmbrus · 2015-04-03T01:19:12Z

ping @liancheng

tianyi · 2015-04-28T02:36:58Z

Since there is a big difference between this PR and the latest master branch, so I have created a new PR #5730 for this feature.
So this PR should be closed.

This PR is a rebased version of #3946 , and mainly focused on creating an independent tab for the thrift server in spark web UI. Features: 1. Session related statistics ( username and IP are only supported in hive-0.13.1 ) 2. List all the SQL executing or executed on this server 3. Provide links to the job generated by SQL 4. Provide link to show all SQL executing or executed in a specified session Prototype snapshots: This is the main page for thrift server ![image](https://cloud.githubusercontent.com/assets/1411869/7361379/df7dcc64-ed89-11e4-9964-4df0b32f475e.png) Author: tianyi <[email protected]> Closes #5730 from tianyi/SPARK-5100 and squashes the following commits: cfd14c7 [tianyi] style fix 0efe3d5 [tianyi] revert part of pom change c0f2fa0 [tianyi] extends HiveThriftJdbcTest to start/stop thriftserver for UI test aa20408 [tianyi] fix style problem c9df6f9 [tianyi] add testsuite for thriftserver ui and fix some style issue 9830199 [tianyi] add webui for thriftserver

This PR is a rebased version of apache#3946 , and mainly focused on creating an independent tab for the thrift server in spark web UI. Features: 1. Session related statistics ( username and IP are only supported in hive-0.13.1 ) 2. List all the SQL executing or executed on this server 3. Provide links to the job generated by SQL 4. Provide link to show all SQL executing or executed in a specified session Prototype snapshots: This is the main page for thrift server ![image](https://cloud.githubusercontent.com/assets/1411869/7361379/df7dcc64-ed89-11e4-9964-4df0b32f475e.png) Author: tianyi <[email protected]> Closes apache#5730 from tianyi/SPARK-5100 and squashes the following commits: cfd14c7 [tianyi] style fix 0efe3d5 [tianyi] revert part of pom change c0f2fa0 [tianyi] extends HiveThriftJdbcTest to start/stop thriftserver for UI test aa20408 [tianyi] fix style problem c9df6f9 [tianyi] add testsuite for thriftserver ui and fix some style issue 9830199 [tianyi] add webui for thriftserver

liancheng reviewed Jan 8, 2015
View reviewed changes

scwf reviewed Jan 8, 2015
View reviewed changes

tianyi added 4 commits January 19, 2015 12:50

add thriftserver-ui support

f1aedeb

fix bugs, remove IP trace due to hive-0.12 not support

3cca2fb

fix some bug for hive-0.12

703e331

fix some issues

daed3d1

tianyi force-pushed the thriftserver-ui branch from 14a461d to daed3d1 Compare January 19, 2015 04:53

fix shim12 rebase issue; make session id as the default groupId

fb507df

liancheng reviewed Jan 19, 2015
View reviewed changes

move SparkSQLSessionManager to Shim.

32332cb

use statementId as groupId to support multiple jobs for one sql. support multiple jobs for one sql in ThriftServerPage.

combine ThriftServerUIEventListener with exists listener in HiveThrif…

fd35262

…tServer2

tianyi closed this Apr 28, 2015

liancheng mentioned this pull request Apr 28, 2015

[SPARK-5100][SQL]add webui for thriftserver #5730

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-5100][SQL] add thriftserver-ui support #3946

[SPARK-5100][SQL] add thriftserver-ui support #3946

tianyi commented Jan 8, 2015

liancheng commented Jan 8, 2015

liancheng Jan 8, 2015

liancheng commented Jan 8, 2015

liancheng Jan 8, 2015

SparkQA commented Jan 8, 2015

liancheng Jan 8, 2015

tianyi Jan 14, 2015

liancheng commented Jan 8, 2015

scwf Jan 8, 2015

tianyi commented Jan 16, 2015

tianyi commented Jan 19, 2015

SparkQA commented Jan 19, 2015

SparkQA commented Jan 19, 2015

SparkQA commented Jan 19, 2015

liancheng Jan 19, 2015

SparkQA commented Jan 20, 2015

scwf commented Jan 21, 2015

liancheng commented Jan 21, 2015

SparkQA commented Jan 26, 2015

marmbrus commented Apr 3, 2015

tianyi commented Apr 28, 2015


		import scala.xml.Node

		/** Page for Spark Web UI that shows statistics of a streaming job */

[SPARK-5100][SQL] add thriftserver-ui support #3946

[SPARK-5100][SQL] add thriftserver-ui support #3946

Conversation

tianyi commented Jan 8, 2015

liancheng commented Jan 8, 2015

liancheng Jan 8, 2015

Choose a reason for hiding this comment

liancheng commented Jan 8, 2015

liancheng Jan 8, 2015

Choose a reason for hiding this comment

SparkQA commented Jan 8, 2015

liancheng Jan 8, 2015

Choose a reason for hiding this comment

tianyi Jan 14, 2015

Choose a reason for hiding this comment

liancheng commented Jan 8, 2015

scwf Jan 8, 2015

Choose a reason for hiding this comment

tianyi commented Jan 16, 2015

tianyi commented Jan 19, 2015

SparkQA commented Jan 19, 2015

SparkQA commented Jan 19, 2015

SparkQA commented Jan 19, 2015

liancheng Jan 19, 2015

Choose a reason for hiding this comment

SparkQA commented Jan 20, 2015

scwf commented Jan 21, 2015

liancheng commented Jan 21, 2015

SparkQA commented Jan 26, 2015

marmbrus commented Apr 3, 2015

tianyi commented Apr 28, 2015