Adding class path with wildcard calls resetEnvironment repeatedly, can be very slow

[jpallas]

If I use the %classpath add jar /path/to/dir/* wildcard mechanism, the default behavior (in BaseEvaluator.addJarToClassPath, called indirectly by MagicCommand.classpathAddJar) will end up calling resetEnvironment once for each jar file.

Now, the Scala interpreter is famously slow to start up, which even the most ardent fans will admit. The consequence in my case was that adding 209 jar files (from the Spark jars directory) blocked my Scala notebook for over ten minutes. I didn't do actual profiling, but every time I ran jstack I saw resetEnvironment in the stack, so I'm fairly confident about the cause.

[scottdraves]

Hi Joe a fix has been merged to master, though I didn't have a good test for it.
Could you confirm it works for you?

[jpallas]

Yes, that works. I can import the spark lib jars and run a simple session in about 35 seconds on my laptop. For the record, here's what my trivial Spark test notebook looks like.

%classpath add jar /usr/local/Cellar/apache-spark/2.2.0/libexec/jars/*

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder.appName("Beaker").master("local").getOrCreate()
import spark.implicits._

val ds = spark.createDataset(Seq(("this", "that"), ("one thing", "another")))

import scala.collection.JavaConverters._
ds.collect.toMap.asJava

spark.stop()

[jpallas]

This seems to have gotten broken by 9d7150b. MagicCommand.addJars is calling kernel.addJarToClassPath instead of kernel.addJarsToClassPath.

[scottdraves]

my fault, should be good now.