Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-3627] - [yarn] - fix exit code and final status reporting to RM #2577

Closed
wants to merge 8 commits into from
Closed

[SPARK-3627] - [yarn] - fix exit code and final status reporting to RM #2577

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
d3cc800
SPARK-3627 - yarn - fix exit code and final status reporting to RM
tgravescs Sep 29, 2014
f0b6519
change order of cleanup staging dir
tgravescs Sep 29, 2014
32f4dfa
switch back
tgravescs Sep 29, 2014
fab166d
update based on review comments
tgravescs Sep 30, 2014
85f1901
Merge remote-tracking branch 'upstream/master' into SPARK-3627
tgravescs Oct 2, 2014
24c98e3
rework
tgravescs Oct 3, 2014
e8cc261
fix accidental typo during fixing comment
tgravescs Oct 3, 2014
9c2efbf
review comments
tgravescs Oct 6, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 16 additions & 10 deletions yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/YarnRMClientImpl.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ private class YarnRMClientImpl(args: ApplicationMasterArguments) extends YarnRMC
private var rpc: YarnRPC = null
private var resourceManager: AMRMProtocol = _
private var uiHistoryAddress: String = _
private var registered: Boolean = false

override def register(
conf: YarnConfiguration,
Expand All @@ -51,8 +52,11 @@ private class YarnRMClientImpl(args: ApplicationMasterArguments) extends YarnRMC
this.rpc = YarnRPC.create(conf)
this.uiHistoryAddress = uiHistoryAddress

resourceManager = registerWithResourceManager(conf)
registerApplicationMaster(uiAddress)
synchronized {
resourceManager = registerWithResourceManager(conf)
registerApplicationMaster(uiAddress)
registered = true
}

new YarnAllocationHandler(conf, sparkConf, resourceManager, getAttemptId(), args,
preferredNodeLocations, securityMgr)
Expand All @@ -66,14 +70,16 @@ private class YarnRMClientImpl(args: ApplicationMasterArguments) extends YarnRMC
appAttemptId
}

override def shutdown(status: FinalApplicationStatus, diagnostics: String = "") = {
val finishReq = Records.newRecord(classOf[FinishApplicationMasterRequest])
.asInstanceOf[FinishApplicationMasterRequest]
finishReq.setAppAttemptId(getAttemptId())
finishReq.setFinishApplicationStatus(status)
finishReq.setDiagnostics(diagnostics)
finishReq.setTrackingUrl(uiHistoryAddress)
resourceManager.finishApplicationMaster(finishReq)
override def unregister(status: FinalApplicationStatus, diagnostics: String = "") = synchronized {
if (registered) {
val finishReq = Records.newRecord(classOf[FinishApplicationMasterRequest])
.asInstanceOf[FinishApplicationMasterRequest]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You probably don't need this cast

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this pr didn't change this code, other then wrapping it with an if. Its also going to be deprecated soon so I don't see a reason to fix it.

finishReq.setAppAttemptId(getAttemptId())
finishReq.setFinishApplicationStatus(status)
finishReq.setDiagnostics(diagnostics)
finishReq.setTrackingUrl(uiHistoryAddress)
resourceManager.finishApplicationMaster(finishReq)
}
}

override def getProxyHostAndPort(conf: YarnConfiguration) =
Expand Down
Loading