[YARN] SPARK-2577: File upload to viewfs is broken due to mount point re...

[gerashegalov]

Opting to the option 2 defined in SPARK-2577, i.e., retrieve and pass the correct file system object to addResource.

[AmplabJenkins]

Can one of the admins verify this patch?

[tgravescs]

Jenkins, test this please

[SparkQA]

QA tests have started for PR 1483. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16833/consoleFull

[SparkQA]

QA results for PR 1483:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16833/consoleFull

[sryza]

@tgravescs do you know whether we need to do anything to fetch tokens for the destination filesystem? Otherwise the change looks good to me.

[tgravescs]

From my understanding when using viewfs: addDelegationTokens is supposed to get tokens for all the underlying filesystems so it should already have a token for it. @gerashegalov did you test this on secure cluster?

[tgravescs]

+1.

[gerashegalov]

@tgravescs Sorry for coming late. I haven't tested on a secure cluster yet, only on my pseudo-distributed cluster. But since the viewfs implements getChildFileSystems, it should not be an issue.