Fix for SI-9881 in branch 2.11.x

[mpetruska]

Retrofits fix for SI-9881 to branch 2.11.x. Contains the changes of #5640 .

---

This change should fix Spark issue 22393

[mpetruska]

Not sure what causes the failure: "combined — Found earlier commit(s) marked failure: 68118b"...

[mpetruska]

Jenkins, re-test this please.

[SethTisue]

/rebuild

[mpetruska]

It seems that 68118b failed due to the fact I did not sign the cla before I created the PR; and 89150d0 fails because the 68118b failed...
Not sure how can we fix this...

[SethTisue]

yeah, looks like a Scabot limitation. we can just /nothingtoseehere it before merging

[SethTisue]

note that in general, we're no longer accepting changes for 2.11.x and there are currently no plans for a 2.11.13 release. you might able to persuade @adriaanm, but you'd probably need to explicitly make a case that this is both sufficiently critical and sufficiently safe

[mpetruska]

Hi, thanks for the info. The bug does not seem to be critical (marked as minor in Jira), so I don't think we should make too much effort to merge/release this one.
Currently looking for alternative fixes for the issue; it seems that the problem only appears if isClassBased is set to true. Will look for a way to force Spark-shell to a non-class based behaviour, if possible.

[adriaanm]

I would be open to a 2.11 release dedicated to fixing bugs that affect Spark (at least a half dozen, to make it worth cutting a release), if someone is willing to step up and coordinate the work. We can't afford to invest in 2.11 anymore, but I won't stand in the way of progress :-) (Wellllll, ideally, the progress would be for Spark to upgrade to 2.12 ;-))

Regarding this bug: the workaround of turning of the class based encoding is unlikely to work because I think Spark relies on it for improved serialization or lower memory retention?

[mpetruska]

Hi @adriaanm,
Thank you for the information and the opportunity.
I looked into Spark issues related to "Spark-shell" and came up with a list of tickets that have some chance of being caused by a bug in the Scala 2.11 repl:

• SPARK-22393 created: 2017/10/17 (confirmed, this is bug cause by the underlying Scala repl)
• SPARK-20706 created: 2017/05/11 (unconfirmed)
• SPARK-10806 created: 2015/09/15 (unconfirmed)
• SPARK-7043 created: 2015/04/15 (unconfirmed)
• SPARK-5150 created: 2015/01/08 (unconfirmed)

Based on this list, I find it highly unprobable to collect the number of changes that would warrant a 2.11 release.
Thanks for the info and the opportunity, though.

[mpetruska]

As discussed above, it does not seem practical to plan a release for these changes.

[adriaanm]

Thanks for digging in, @mpetruska! We are still considering another 2.11 with some other small fixes, so we could try to tackle at least a few of the spark shell issues that have a low chance of causing regressions. Track scala/scala-dev#451 for the planning around it. I expect a release either late this year or beginning of January.

[mpetruska]

Re-opening to allow tracking the changes in scala/scala-dev#451.

[SethTisue]

labeling as WIP so it doesn't show up when filter on that label's absence

[adriaanm]

Tested locally. TODO: bring back CI for 2.11.x if we're going to do cut another release.

[SethTisue]

this is failing on Windows at https://scala-ci.typesafe.com/view/scala-2.11.x/job/scala-2.11.x-integrate-windows/

[jpallas]

Is … is this complaining about white space (maybe Windows line-ending issue)?

% diff Y:\jenkins\workspace\scala-2.11.x-integrate-windows\test\files\run\t9880-9881-run.log Y:\jenkins\workspace\scala-2.11.x-integrate-windows\test\files\run\t9880-9881.checkdiff Y:\jenkins\workspace\scala-2.11.x-integrate-windows\test\files\run\t9880-9881-run.log Y:\jenkins\workspace\scala-2.11.x-integrate-windows\test\files\run\t9880-9881.check
--- t9880-9881.check
+++ t9880-9881-run.log
@@ -15,3 +15,3 @@
 
-scala>
+scala> 
 
@@ -28,3 +28,3 @@
 
-scala>
+scala> 
 

[SethTisue]

very likely Windows line endings

[SethTisue]

attempting a fix over at #6777