[SPARK-2750][WEB UI]Add Https support for Web UI #5664

WangTaoTheTonic · 2015-04-23T12:46:01Z

Based on #1980, as @scwf is busy doing sql stuff, I will finish this patch.

SparkQA · 2015-04-23T12:51:12Z

Test build #30839 has finished for PR 5664 at commit fef5003.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

scwf · 2015-04-23T13:04:09Z

sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala

@@ -1,571 +0,0 @@
-/*


revert this

SparkQA · 2015-04-23T13:11:16Z

Test build #30841 has finished for PR 5664 at commit d80f7e9.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- protected[sql] abstract class AtomicType extends DataType
- abstract class NumericType extends AtomicType
- class Encoder[T <: AtomicType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
- class Decoder[T <: AtomicType](buffer: ByteBuffer, columnType: NativeColumnType[T])
- class Encoder[T <: AtomicType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
- class Decoder[T <: AtomicType](buffer: ByteBuffer, columnType: NativeColumnType[T])
- class Encoder[T <: AtomicType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
- class Decoder[T <: AtomicType](buffer: ByteBuffer, columnType: NativeColumnType[T])
This patch does not change any dependencies.

SparkQA · 2015-05-04T09:48:39Z

Test build #31749 has finished for PR 5664 at commit 18982b4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

WangTaoTheTonic · 2015-05-04T10:45:46Z

@vanzin
Thanks for you advise. After importing my cert into JVM ResourceManager use, I can jump to the driver UI page from RM's link.

After taking some look at the docs and codes, I rename the configs to spark.ssl.ui.* so that it can be set via spark.ssl.* by default.

@jacek-lewandowski Could you help to review yet?

vanzin · 2015-05-04T18:11:26Z

core/src/main/scala/org/apache/spark/SecurityManager.scala


  logDebug(s"SSLConfiguration for file server: $fileServerSSLOptions")
  logDebug(s"SSLConfiguration for Akka: $akkaSSLOptions")
+  logDebug(s"SSLConfiguration for Akka: $webUISSLOptions")


SparkQA · 2015-05-05T03:19:52Z

Test build #31824 has finished for PR 5664 at commit dfbe1d6.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

WangTaoTheTonic · 2015-05-05T04:48:19Z

Jenkins, retest this please.

WangTaoTheTonic · 2015-05-05T06:28:52Z

@vanzin Not sure if we should modify createAkkaConfig, please give some reference.

SparkQA · 2015-05-05T06:39:34Z

Test build #31831 has finished for PR 5664 at commit dfbe1d6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2015-05-05T16:56:51Z

I'm not really familiar with Akka's SSL configuration, don't even know if they support those options. But if not modifying it, you should mention in the docs that the options don't apply to Akka.

WangTaoTheTonic · 2015-05-06T01:12:10Z

@CodingCat @jacek-lewandowski We add some configs for SSLOptions (keystoreType, needClientAuth, trustStoreType), they are supported in Jetty SSL, but we have no idea if akka support them too. Could you guys give some thought?

@pwendell @andrewor14 @srowen Do you know anybody else who is familiar with Akka'\s SSL configurations?

andrewor14 · 2015-09-02T02:20:22Z

@vanzin @JoshRosen is this already resolved by #3571?

vanzin · 2015-09-02T02:26:39Z

I don't think so; from what I remember, that PR only added SSL to the HTTP server that serves jars and files to executors, not to the server that serves the UI.

jacek-lewandowski · 2015-09-02T04:27:49Z

@vanzin @andrewor14 I may take a look at this and finish or create a separate pr with my approach if you like.

steveloughran · 2015-09-10T11:14:04Z

I'm not sure that YARN likes custom SSL certificates at all. Even if you do set up the RM proxy up for them, there's the issue that as they are host-wide, so every web/REST endpoint on the host can/should use the same certificate. That said, there's probably no explicit reason why the RM Proxy doesn't handle SSL except it's not been done —that'd be the place to add the code.

Note of course that web/REST clients of AM web pages redirected via the Am filter will have their communications direct with that RM proxy, using SPNEGO and any https support of the AM proxy; you can lock down the long-haul bit of the communications without adding HTTPS. It's the in-cluster TCP connections that HTTPS -in-AM would address

jacek-lewandowski · 2015-10-29T14:18:21Z

@WangTaoTheTonic can you rebase and squash?

WangTaoTheTonic · 2015-10-30T06:37:28Z

@jacek-lewandowski Okay I will rebase this in a week or so.

jacek-lewandowski · 2015-10-30T11:28:49Z

@WangTaoTheTonic if you don't have time, would you mind if i take your commits, rebase, squash them and add few changes?

WangTaoTheTonic · 2015-10-31T02:15:05Z

@jacek-lewandowski Sure. I'm glad for this.

andrewor14 · 2015-11-10T19:41:30Z

@WangTaoTheTonic would you mind closing this patch for now? We can always re-open it later if there is interest or if @jacek-lewandowski takes over.

jacek-lewandowski · 2015-11-12T12:49:28Z

I rebased this pr on the current master with a little pain 😄
Will create a new PR soon.

WangTaoTheTonic · 2015-11-13T06:30:50Z

ok once jacky raise the PR, I will close this one.

pritpalm · 2016-12-15T16:05:28Z

I want to enable https on spark UI. I added following config to spark-defaults.config, but when we access spark ui via https::/:8080 or https://:443 or https://:8480, it's not able to connect.

spark.ui.https.enabled true
spark.ssl.keyPassword abcd
spark.ssl.keyStore rtmqa-clientid.jks
spark.ssl.keyStorePassword changeit

scwf added 24 commits August 16, 2014 08:20

support https in spark web ui

e8256e5

add unit test

8f7cc96

fix according to comments

c90d84e

fix scalastyle

de8d1bd

fix workerinfo in JsonProtocolSuite

35074fd

Merge branch 'master' of github.com:apache/spark into https

967950f

import org.eclipse.jetty.server.Server to fix test error

9591c9c

fix comments

333334a

add redirect from http to https

64d7dc0

revert debug code

89bf986

merge with apache/master

8e5132d

add https/ssl to docs

677b746

fix docs

a4ce923

fix code format

6c31dc7

fix securePort

7a898fb

fix conflict with apache/master

a29ec86

merge with apache/master and fix conflict

55bbc5f

fix comments by JoshRosen

e5c87cb

fix conflict

baaa1ce

address JoshRosen's comments

a48c6fc

address vanzin's comments

2dadb2f

Merge branch 'master' of https://github.com/apache/spark into https

8b32853

add reference to method newURI

3b01d3a

Merge branch 'master' of https://github.com/apache/spark into https

4ae834b

scwf mentioned this pull request Apr 23, 2015

[SPARK-2750] support https in spark web ui #1980

Closed

scwf reviewed Apr 23, 2015
View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala

@@ -1,571 +0,0 @@

/*

Copy link

Contributor

scwf Apr 23, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert this

rebase based on apache#1980

d80f7e9

WangTaoTheTonic force-pushed the SPARK-2750 branch from fef5003 to d80f7e9 Compare April 23, 2015 13:06

WangTaoTheTonic changed the title ~~[SPARK-2750][WIP]Add Https support for Web UI~~ [SPARK-2750][WEB UI]Add Https support for Web UI May 4, 2015

vanzin reviewed May 4, 2015
View reviewed changes

per Marcelo's comments

dfbe1d6

vanzin mentioned this pull request Dec 10, 2015

[SPARK-2750][WEB UI] Add https support to the Web UI #10238

Closed

asfgit closed this in ce5fd40 Dec 17, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-2750][WEB UI]Add Https support for Web UI #5664

[SPARK-2750][WEB UI]Add Https support for Web UI #5664

WangTaoTheTonic commented Apr 23, 2015

SparkQA commented Apr 23, 2015

scwf Apr 23, 2015

SparkQA commented Apr 23, 2015

SparkQA commented May 4, 2015

WangTaoTheTonic commented May 4, 2015

vanzin May 4, 2015

SparkQA commented May 5, 2015

WangTaoTheTonic commented May 5, 2015

WangTaoTheTonic commented May 5, 2015

SparkQA commented May 5, 2015

vanzin commented May 5, 2015

WangTaoTheTonic commented May 6, 2015

andrewor14 commented Sep 2, 2015

vanzin commented Sep 2, 2015

jacek-lewandowski commented Sep 2, 2015

steveloughran commented Sep 10, 2015

jacek-lewandowski commented Oct 29, 2015

WangTaoTheTonic commented Oct 30, 2015

jacek-lewandowski commented Oct 30, 2015

WangTaoTheTonic commented Oct 31, 2015

andrewor14 commented Nov 10, 2015

jacek-lewandowski commented Nov 12, 2015

WangTaoTheTonic commented Nov 13, 2015

pritpalm commented Dec 15, 2016

[SPARK-2750][WEB UI]Add Https support for Web UI #5664

[SPARK-2750][WEB UI]Add Https support for Web UI #5664

Conversation

WangTaoTheTonic commented Apr 23, 2015

SparkQA commented Apr 23, 2015

scwf Apr 23, 2015

Choose a reason for hiding this comment

SparkQA commented Apr 23, 2015

SparkQA commented May 4, 2015

WangTaoTheTonic commented May 4, 2015

vanzin May 4, 2015

Choose a reason for hiding this comment

SparkQA commented May 5, 2015

WangTaoTheTonic commented May 5, 2015

WangTaoTheTonic commented May 5, 2015

SparkQA commented May 5, 2015

vanzin commented May 5, 2015

WangTaoTheTonic commented May 6, 2015

andrewor14 commented Sep 2, 2015

vanzin commented Sep 2, 2015

jacek-lewandowski commented Sep 2, 2015

steveloughran commented Sep 10, 2015

jacek-lewandowski commented Oct 29, 2015

WangTaoTheTonic commented Oct 30, 2015

jacek-lewandowski commented Oct 30, 2015

WangTaoTheTonic commented Oct 31, 2015

andrewor14 commented Nov 10, 2015

jacek-lewandowski commented Nov 12, 2015

WangTaoTheTonic commented Nov 13, 2015

pritpalm commented Dec 15, 2016