[SQL] Various DataFrame DSL update.

[rxin]

1. Added foreach, foreachPartition, flatMap to DataFrame.
2. Added col() in dsl.
3. Support renaming columns in toDataFrame.
4. Support type inference on arrays (in addition to Seq).
5. Updated mllib to use the new DSL.

[rxin]

cc @mengxr for mllib changes

[SparkQA]

Test build #26285 has started for PR 4260 at commit 62608c4.

• This patch merges cleanly.

[SparkQA]

Test build #26285 has finished for PR 4260 at commit 62608c4.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26285/
Test FAILed.

[SparkQA]

Test build #26287 has started for PR 4260 at commit d31fcd2.

• This patch merges cleanly.

[SparkQA]

Test build #26294 has started for PR 4260 at commit fab3ccc.

• This patch merges cleanly.

[SparkQA]

Test build #26287 has finished for PR 4260 at commit d31fcd2.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26287/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26294/
Test FAILed.

[shaneknapp]

i will kick this off again once i restart jenkins.

[shaneknapp]

jenkins, test this please.

[shaneknapp]

jenkins, test this please.

[SparkQA]

Test build #26299 has started for PR 4260 at commit 73466c1.

• This patch merges cleanly.

[mengxr]

minor: The word col might be used as matrix column index in ML algorithms.

This line is still not straightforward to read. I'm thinking of something like the following

val scoreFunc = UDF((score: Double) => {if (score > t) 1.0 else 0.0})
dataset.select($"*", scoreFunc(col(map(featuresCol))).as(map(scoreCol))

[mengxr]

@rxin The ALS join code is much easier to read now:) I hope UDFs can be used as functions instead of an argument of callUDF. Besides, it would be nice to interpret strings as columns by default and force using Literal for constant strings. (I'm limited by the scope of ML usages.) Those could be done in follow-up PRs. The ML changes in this PR look good to me.

[SparkQA]

Test build #26299 has finished for PR 4260 at commit 73466c1.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26299/
Test FAILed.