[SPARK-7403][WebUI] Link URL in objects on Timeline View is wrong in case of running on YARN #5947

sarutak · 2015-05-06T18:59:41Z

When we use Spark on YARN and have AllJobPage via ResourceManager's proxy, the link URL in objects which represent each job on timeline view is wrong.

In timeline-view.js, the link is generated as follows.

window.location.href = "job/?id=" + getJobId(this);

This assumes the URL displayed on the web browser ends with "jobs/" but when we access AllJobPage via the proxy, the url displayed does not end with "jobs/"

The proxy doesn't return status code 301 or 302 so the url displayed still indicates the base url, not "/jobs" even though displaying AllJobPages.

AmplabJenkins · 2015-05-06T19:02:11Z

Merged build triggered.

AmplabJenkins · 2015-05-06T19:02:20Z

Merged build started.

SparkQA · 2015-05-06T19:02:45Z

Test build #32015 has started for PR 5947 at commit 860faa7.

SparkQA · 2015-05-06T20:06:09Z

Test build #32015 has finished for PR 5947 at commit 860faa7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class Model(Transformer):
- class PipelineModel(Model):
- class CrossValidator(Estimator):
- class CrossValidatorModel(Model):
- class JavaModel(Model, JavaTransformer):
- class JoinedRow6 extends Row
- case class WindowSpecDefinition(
- case class WindowSpecReference(name: String) extends WindowSpec
- sealed trait FrameBoundary
- case class ValuePreceding(value: Int) extends FrameBoundary
- case class ValueFollowing(value: Int) extends FrameBoundary
- case class SpecifiedWindowFrame(
- trait WindowFunction extends Expression
- case class UnresolvedWindowFunction(
- case class UnresolvedWindowExpression(
- case class WindowExpression(
- case class WithWindowDefinition(
- case class Window(
- case class Window(
- case class ComputedWindow(

AmplabJenkins · 2015-05-06T20:06:14Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-06T20:06:14Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32015/
Test FAILed.

andrewor14 · 2015-05-06T23:29:57Z

retest this please

andrewor14 · 2015-05-06T23:30:06Z

core/src/main/resources/org/apache/spark/ui/static/timeline-view.css

@@ -19,7 +19,8 @@ div#application-timeline, div#job-timeline {
  margin-bottom: 30px;
 }

-#application-timeline div.legend-area {
+#application-timeline div.legend-area ,


no space before ,

andrewor14 · 2015-05-06T23:30:50Z

Thanks @sarutak LGTM.

andrewor14 · 2015-05-06T23:32:02Z

By the way, it's a separate issue but when I open the JS console I see some errors with viz.js

Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost:5050/static/dist/vis.js

Do you know what's wrong?

AmplabJenkins · 2015-05-06T23:32:10Z

Merged build triggered.

AmplabJenkins · 2015-05-06T23:32:17Z

Merged build started.

SparkQA · 2015-05-06T23:32:51Z

Test build #32050 has started for PR 5947 at commit 860faa7.

SparkQA · 2015-05-06T23:46:31Z

Test build #32050 has finished for PR 5947 at commit 860faa7.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-06T23:46:34Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-06T23:46:35Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32050/
Test FAILed.

AmplabJenkins · 2015-05-07T02:12:10Z

Merged build triggered.

AmplabJenkins · 2015-05-07T02:12:18Z

Merged build started.

SparkQA · 2015-05-07T02:14:16Z

Test build #32066 has started for PR 5947 at commit bda684e.

sarutak · 2015-05-07T02:14:31Z

@andrewor14 Thanks for letting me know another issue. I found the reason is vis.js (not vis.min.js) is absent. vis.js is designated in vis.map which is a file for Source Maps.

Source Maps is used for symbol resolution for abbreviated symbols in *.min.js when we debug (http://www.html5rocks.com/en/tutorials/developertools/sourcemaps/).

vis.map and vis.js are only downloaded when we use debug tools embedded in browsers.

SparkQA · 2015-05-07T04:02:20Z

Test build #32066 has finished for PR 5947 at commit bda684e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-07T04:02:25Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-07T04:02:25Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32066/
Test PASSed.

srowen · 2015-05-07T08:27:11Z

.rat-excludes

@@ -35,6 +35,7 @@ dagre-d3.min.js
 graphlib-dot.min.js
 sorttable.js
 vis.min.js
+vis.js


Why is this needed now?

It's because vis.js is referred by vis.map and vis.map is referred by vis.min.js.
vis.js is only downloaded when Source Maps is enabled for debug.
Firefox or Chrome support SourceMaps and browsers which don't support Source Maps just ignore the reference directive in vis.min.js.

Why does that affect the Spark source tree though? Maybe dumb question

I think it's because we can't expect the cluster to have network access to the external world so we need to bundle it.

But @sarutak what do we use vis.map for? The timeline seems to work fine even without vis.js. What are the specific UI differences?

I think it's just for debuging. See https://developer.chrome.com/devtools/docs/javascript-debugging

Search "Using Source Maps" in this page

But I think we don't need to include "vis.js" in Spark since it's just for debuging. The developers can add them by themselves locally.

In that case I would prefer to leave it out if possible. This is bringing in a 30k lines file into our repo, not to mention that it slows down the page load speed.

zsxwing · 2015-05-07T19:24:07Z

@sarutak could you also add vis to the LICENSE file, please?

zsxwing · 2015-05-07T19:26:08Z

@sarutak this issue only happens in the index page. In other pages, we can use the relative path. Right?

andrewor14 · 2015-05-07T19:43:21Z

@sarutak sorry I wasn't being clear. I would prefer that this patch only address the YARN issue so we can merge it more easily. We should address the missing vis.js error separately (either by removing usages of it, which I prefer, or pulling in the needed dependency).

sarutak · 2015-05-07T22:51:16Z

I think Source Maps doesn't affect page load speed because non-compressed vis.js is only downloaded when we debug (using developer tools of Firefox or like that) but there is some reason in what you says.

Anyway, I'll remove vis.js from my changes for now.
Tnanks @andrewor14 ,@srowen and @zsxwing !

sarutak · 2015-05-07T22:54:57Z

@zsxwing Yes, I think it happens only index page of Web UI via ResourceManager's proxy (HistoryServer doesn't have this issue).

AmplabJenkins · 2015-05-07T23:47:19Z

Merged build started.

SparkQA · 2015-05-07T23:49:25Z

Test build #32164 has started for PR 5947 at commit b1052e7.

SparkQA · 2015-05-08T01:31:43Z

Test build #32164 has finished for PR 5947 at commit b1052e7.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class ElementwiseProduct extends UnaryTransformer[Vector, Vector, ElementwiseProduct]
- class ElementwiseProduct(val scalingVector: Vector) extends VectorTransformer

AmplabJenkins · 2015-05-08T01:31:48Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-08T01:31:48Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32164/
Test PASSed.

AmplabJenkins · 2015-05-09T04:32:10Z

Merged build triggered.

AmplabJenkins · 2015-05-09T04:32:16Z

Merged build started.

SparkQA · 2015-05-09T04:33:58Z

Test build #32293 has started for PR 5947 at commit aaf40e1.

SparkQA · 2015-05-09T05:40:02Z

Test build #32293 has finished for PR 5947 at commit aaf40e1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-09T05:40:07Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-09T05:40:08Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32293/
Test FAILed.

sarutak · 2015-05-09T05:44:13Z

retest this please.

AmplabJenkins · 2015-05-09T05:47:10Z

Merged build triggered.

AmplabJenkins · 2015-05-09T05:47:16Z

Merged build started.

SparkQA · 2015-05-09T05:49:06Z

Test build #32298 has started for PR 5947 at commit aaf40e1.

SparkQA · 2015-05-09T07:33:40Z

Test build #32298 has finished for PR 5947 at commit aaf40e1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-09T07:33:45Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-09T07:33:45Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32298/
Test PASSed.

… case of running on YARN When we use Spark on YARN and have AllJobPage via ResourceManager's proxy, the link URL in objects which represent each job on timeline view is wrong. In timeline-view.js, the link is generated as follows. ``` window.location.href = "job/?id=" + getJobId(this); ``` This assumes the URL displayed on the web browser ends with "jobs/" but when we access AllJobPage via the proxy, the url displayed does not end with "jobs/" The proxy doesn't return status code 301 or 302 so the url displayed still indicates the base url, not "/jobs" even though displaying AllJobPages. ![2015-05-07 3 34 37](https://cloud.githubusercontent.com/assets/4736016/7501079/a8507ad6-f46c-11e4-9bed-62abea170f4c.png) Author: Kousuke Saruta <[email protected]> Closes #5947 from sarutak/fix-link-in-timeline and squashes the following commits: aaf40e1 [Kousuke Saruta] Added Copyright for vis.js 01bee7b [Kousuke Saruta] Fixed timeline-view.js in order to get correct href (cherry picked from commit 12b95ab) Signed-off-by: Sean Owen <[email protected]>

… case of running on YARN When we use Spark on YARN and have AllJobPage via ResourceManager's proxy, the link URL in objects which represent each job on timeline view is wrong. In timeline-view.js, the link is generated as follows. ``` window.location.href = "job/?id=" + getJobId(this); ``` This assumes the URL displayed on the web browser ends with "jobs/" but when we access AllJobPage via the proxy, the url displayed does not end with "jobs/" The proxy doesn't return status code 301 or 302 so the url displayed still indicates the base url, not "/jobs" even though displaying AllJobPages. ![2015-05-07 3 34 37](https://cloud.githubusercontent.com/assets/4736016/7501079/a8507ad6-f46c-11e4-9bed-62abea170f4c.png) Author: Kousuke Saruta <[email protected]> Closes apache#5947 from sarutak/fix-link-in-timeline and squashes the following commits: aaf40e1 [Kousuke Saruta] Added Copyright for vis.js 01bee7b [Kousuke Saruta] Fixed timeline-view.js in order to get correct href

andrewor14 reviewed May 6, 2015
View reviewed changes

srowen reviewed May 7, 2015
View reviewed changes

sarutak force-pushed the fix-link-in-timeline branch from bda684e to 5d91d0d Compare May 7, 2015 23:42

sarutak mentioned this pull request May 8, 2015

[WebUI] Remove debug feature for vis.js #5994

Closed

sarutak added 2 commits May 9, 2015 13:29

Fixed timeline-view.js in order to get correct href

01bee7b

Added Copyright for vis.js

aaf40e1

sarutak force-pushed the fix-link-in-timeline branch from b1052e7 to aaf40e1 Compare May 9, 2015 04:30

asfgit closed this in 12b95ab May 9, 2015

sarutak deleted the fix-link-in-timeline branch May 18, 2015 01:52

Tagar mentioned this pull request Mar 6, 2017

ZEPPELIN-1692: Ability to access Spark jobs UI from the paragraph apache/zeppelin#1663

Closed

1 task

[SPARK-7403][WebUI] Link URL in objects on Timeline View is wrong in case of running on YARN #5947

[SPARK-7403][WebUI] Link URL in objects on Timeline View is wrong in case of running on YARN #5947

Conversation

sarutak commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

SparkQA commented May 6, 2015

SparkQA commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

andrewor14 commented May 6, 2015

Choose a reason for hiding this comment

andrewor14 commented May 6, 2015

andrewor14 commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

SparkQA commented May 6, 2015

SparkQA commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

SparkQA commented May 7, 2015

sarutak commented May 7, 2015

SparkQA commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zsxwing commented May 7, 2015

zsxwing commented May 7, 2015

andrewor14 commented May 7, 2015

sarutak commented May 7, 2015

sarutak commented May 7, 2015

AmplabJenkins commented May 7, 2015

SparkQA commented May 7, 2015

SparkQA commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 9, 2015

SparkQA commented May 9, 2015

SparkQA commented May 9, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 9, 2015

sarutak commented May 9, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 9, 2015

SparkQA commented May 9, 2015

SparkQA commented May 9, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 9, 2015