Revert "[SPARK-24418][BUILD] Upgrade Scala to 2.11.12 and 2.12.6"

[dbtsai]

This reverts commit c7967c6.

Because of the printWelcome message regression described in https://issues.apache.org/jira/browse/SPARK-25138, to provide our users best experience, @gatorsmile and I agree that we either revert the Scala upgrade or merge #21749 which uses reflection to fix the printing ordering.

I'm leaning toward to merge #21749 since I worry reverting the Scala upgrade will break the work for Scala 2.12. On the other hand, fixing the printing ordering using reflection is not clean.

[srowen]

Oh, what went wrong @dbtsai ?

[rxin]

Can you add to the pr description why we are reverting? Just copy paste what you had above. Thanks.

[gatorsmile]

Thank you! @dbtsai Let us see whether this can pass all the tests?

[srowen]

I see, doesn't look like we lose a critical fix or improvement with the down-grade, and this wasn't released in 2.3.x. OK. I prefer to fix-forward, too, but am neutral on whether the cost of the workaround is worth it.

But does it affect 2.12? yeah that would be a bigger deal. But the original commit didn't seem to touch the 2.12 REPL.

[dbtsai]

@gatorsmile we also need to run the tests for 2.12 manually.

@srowen This PR will revert Scala 2.12.6 back to 2.12.4 since 2.12.6 also removes the hacks we used to initialize Spark. I worry it will have a negative impact on the recent work of Scala 2.12 support.

[SparkQA]

Test build #94977 has finished for PR 22160 at commit dfa3cb5.

• This patch fails Spark unit tests.
• This patch does not merge cleanly.
• This patch adds the following public classes (experimental):
• class SparkILoopInterpreter(settings: Settings, out: JPrintWriter) extends IMain(settings, out)

[SparkQA]

Test build #94978 has finished for PR 22160 at commit 1ad5d3d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class SparkILoopInterpreter(settings: Settings, out: JPrintWriter) extends IMain(settings, out)

[gatorsmile]

@dbtsai Have you tried to run it in scala 2.12? We still can do the upgrade after Apache 2.4 RC.

[srowen]

Oh I get your point @dbtsai -- do you need to revert anything related to 2.12? I think we do need 2.12.6 for fixes for 2.12 to work, but, then again this support isn't released yet. So I figure there's nothing to 'fix' by reverting this?

[dbtsai]

Since 2.12.6 removes the hacks we were using to initialize Spark context, if we want to revert SPARK-24418, we have to use older version of 2.12.x

My 2 cents, reverting SPARK-24418 will make 2.12 work harder since we have to deal with Scala Shell issue as part of the tasks.

Alternatively, instead of reverting SPARK-24418, we should consider to merge #21749 which fixes the message printing issue.

[dbtsai]

@gatorsmile I have not find a time yesterday to run tests with Scala 2.12. I'll try to do it today.

[srowen]

By the way the 2.12 build will still fail, so that won't tell you much.

I think we need 2.12.6 to fix a Java 8 related issue? Don't have it handy though.

Sorry if I'm being dense here but didn't all those REPL changes just concern 2.11? I did not see any 2.12 change except the version

[dbtsai]

@srowen We were overwriting def loadFiles(settings: Settings) in https://github.com/scala/scala/blob/v2.11.6/src/repl/scala/tools/nsc/interpreter/ILoop.scala#L853 to initialize the Spark context.

It's not available in newer version of Scala such as 2.12.6

The REPL code are shared by 2.11 and 2.12, right? So reverting the REPL code will fail 2.12 build.

[dbtsai]

@srowen Just found Scala 2.12 has a separated REPL implementation which suffered from message order printing issue too.

Let me change this PR so the Scala 2.12 version is still on 2.12.6

BTW, the approach in #21749 should work for both 2.12 and 2.11, so we can consolidate the code.

[srowen]

Yeah that's right, we have had two parallel trees for different Scala versions just for reasons like this. There is less forked code now than with 2.10 vs 2.11. If you think we can merge all the code into one tree with some reflection then I think that's really worth it rather than reverting this.

[dbtsai]

Closed this PR since we're in favor of #21749