loadFiles() in ILoop was removed in Scala 2.11.12 which breaks Apache Spark

[dbtsai]

We're migrating to Scala 2.11.12 for JDK9+ support, and we found that loadFIles() in ILoop was removed in Scala 2.11.12.

We use it as a hack to initialize the Spark before REPL sees any files.

https://github.com/apache/spark/blob/master/repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoop.scala#L109

Is it possible to provide a hook for this?

Thanks.

[som-snytt]

Unfortunately, loadInitFiles is a local function. But you can do what sbt console does, and run your init during your override of createInterpreter. The downside for sbt users is that $intp is not available; if that is deemed useful, you can add that, too; ILoop checks isSbt to see how to handle errors at that point. It's safe to intp.initializeSynchronous.

Alternatively, you can externalize the init commands and set the system property scala.repl.autoruncode, which is read in before those -i user files. I don't know that anyone uses the system property. You could check in createInterpreter if it is set, and then merge it with your init in a tmp file. That might be more worthy to be called hack among hackers.

[dbtsai]

Thanks for the reply.

I tried two of the ideas you pointed out without success.

1. Override createInterpreter, but it raises NullPointerException because $intp is not available yet.

2. Initialize the Spark in initializeSynchronous by

class SparkILoop(in0: Option[BufferedReader], out: JPrintWriter)
    extends ILoop(in0, out) {
  ...
  override def createInterpreter(): Unit = {
    intp = new SparkILoopInterpreter(settings, out, initializeSpark)
  }

  def initializeSpark() {
    intp.beQuietDuring {
      savingReplayStack { // remove the commands from session history.
        initializationCommands.foreach(processLine)
      }
    }
  }
  ...
}

and

class SparkILoopInterpreter(settings: Settings, out: JPrintWriter, sparkInitFun: () => Unit)
  extends IMain(settings, out) {
  self =>

  override def initializeSynchronous(): Unit = {
    super.initializeSynchronous()
    sparkInitFun()
  }
  ...
}

this will hang forever.

Can you elaborate more on the alternative option?

Thanks.

[som-snytt]

If you need $intp in the init files, and who doesn't, have you tried: https://github.com/scala/scala/blob/2.12.x/src/repl/scala/tools/nsc/interpreter/ILoop.scala#L996

I'll try to get involved between now and the weekend, it's been on my bucket list.

[dbtsai]

FYI, the original Spark REPL runs initializeSpark() after loopPostInit() and before loadFiles(). Seems in Scala 2.11.12, it's kind of impossible unless scala.repl.autoruncode can do this work. Thanks.

[dbtsai]

@som-snytt are you saying calling intp.quietBind(NamedParam[IMain]("$intp", intp)(tagOfIMain, classTag[IMain])) in def createInterpreter(): Unit to get a $intp? Thanks.

[som-snytt]

@dbtsai Apologies, I ran out of time this weekend because of IKEA. I think I did mean to bind $intp early if you need it, but I haven't tried it yet. Sorry I don't have something better to say. I'll try again when the opportunity presents itself.

[dbtsai]

@som-snytt I got it working in apache/spark#21495

I have couple questions asked there, and would like to get your feedback. Thanks.

[som-snytt]

@dbtsai That's fantabulous! I commented there. The upshot is, after deprecating it, because it wasn't used, I have to revive -Yrepl-sync. Long live -Yrepl-sync!

[srowen]

@dbtsai et al -- see https://issues.apache.org/jira/browse/SPARK-20395 and apache/spark#17982 . The internal change in question happened about 2.11.10 or so if it matters. Reenabling the hack is great I suppose but Spark was always using internal methods that weren't exposed.

[SethTisue]

should this ticket be closed now that apache/spark#21495 has gone through?

[dbtsai]

Thank you all for helping on this!