[SPARK-8461] [SQL] fix codegen with REPL class loader

[davies]

The ExecutorClassLoader for REPL will cause Janino failed to find class for those in java.lang, so switch to use default class loader for Janino, which will also help performance.

cc @liancheng @yhuai

[SparkQA]

Test build #35242 has finished for PR 6898 at commit 7f5ffbe.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #932 has finished for PR 6898 at commit 7f5ffbe.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[lianhuiwang]

i think we do not need to set setContextClassLoader. and we can set ClassBodyEvaluator's classLoader. code is:
val classBodyEval =new ClassBodyEvaluator()
classBodyEval.setParentClassLoader(getClass.getClassLoader)
classBodyEval.cook(code)
classBodyEval.getClazz()
some information: http://janino.net/javadoc/org/codehaus/janino/ClassBodyEvaluator.html,
https://github.com/codehaus/janino/blob/6e61d5b71b4e8682a3497cb1e33a005bee4ca0a2/janino/src/org/codehaus/janino/SimpleCompiler.java#L172

[SparkQA]

Test build #933 has finished for PR 6898 at commit 7f5ffbe.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

Can we add a test in ReplSuite (REPL's tests already has sql core as a dependency)?

[yhuai]

One question that may not related to this pr. If we define a udf in the repl (or user use --jars add a jar containing a udf) and later when we generate code at executor side that calls this udf, can we still find the class?

[davies]

@yhuai The UDF will not be part of generated Java code, so Janino doesn't need to import it. UDF will fallback to call eval(), because UDF can not be presented in Java source.

[SparkQA]

Test build #35273 has finished for PR 6898 at commit 4ff0457.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • abstract class GeneratedClass

[SparkQA]

Test build #35276 has finished for PR 6898 at commit 24276d4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • abstract class GeneratedClass

[liancheng]

Should we also add a REPL test case which uses HiveContext to ensure the isolated classloader plays well with code generation? I know that HiveContext only exists on driver side while code generation happens on executor side, what I'm thinking is there might be chances that they shoot at each other in local mode where everything happens in a single JVM. This can be done in a separate PR since this is a hot fix.

Otherwise looks good, but I'm not super familiar with the whole code generation stuff though.

[davies]

@liancheng Thanks for pointing this out, test always come with costs (time to run), so I'd like delay this until when we really need it. After this patch, Janino only use the default class loader, so it shouldn't be a problem.

Merging this into master as a hotfix!