[SPARK-7407][MLLIB] use uid + name to identify parameters #6019

mengxr · 2015-05-08T22:24:33Z

A param instance is strongly attached to an parent in the current implementation. So if we make a copy of an estimator or a transformer in pipelines and other meta-algorithms, it becomes error-prone to copy the params to the copied instances. In this PR, a param is identified by its parent's UID and the param name. So it becomes loosely attached to its parent and all its derivatives. The UID is preserved during copying or fitting. All components now have a default constructor and a constructor that takes a UID as input. I keep the constructors for Param in this PR to reduce the amount of diff and moved parent as a mutable field.

This PR still needs some clean-ups, and there are several spark.ml PRs pending. I'll try to get them merged first and then update this PR.

@jkbradley

AmplabJenkins · 2015-05-08T22:27:12Z

Merged build triggered.

AmplabJenkins · 2015-05-08T22:27:19Z

Merged build started.

SparkQA · 2015-05-08T22:29:15Z

Test build #32277 has started for PR 6019 at commit a4794dd.

SparkQA · 2015-05-08T22:34:37Z

Test build #32277 has finished for PR 6019 at commit a4794dd.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-08T22:34:38Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-08T22:34:39Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32277/
Test FAILed.

AmplabJenkins · 2015-05-09T00:27:11Z

Build triggered.

AmplabJenkins · 2015-05-09T00:27:19Z

Build started.

SparkQA · 2015-05-09T00:29:35Z

Test build #32285 has started for PR 6019 at commit aa4a611.

SparkQA · 2015-05-09T02:12:27Z

Test build #32285 has finished for PR 6019 at commit aa4a611.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
- class Pipeline(override val uid: String) extends Estimator[PipelineModel]
- final class DecisionTreeClassifier(override val uid: String)
- final class GBTClassifier(override val uid: String)
- class LogisticRegression(override val uid: String)
- final class RandomForestClassifier(override val uid: String)
- class BinaryClassificationEvaluator(override val uid: String)
- final class Binarizer(override val uid: String)
- class ElementwiseProduct(override val uid: String)
- class HashingTF(override val uid: String) extends UnaryTransformer[Iterable[_], Vector, HashingTF]
- final class IDF(override val uid: String) extends Estimator[IDFModel] with IDFBase
- class Normalizer(override val uid: String) extends UnaryTransformer[Vector, Vector, Normalizer]
- class OneHotEncoder(override val uid: String)
- class PolynomialExpansion(override val uid: String)
- class StandardScaler(override val uid: String) extends Estimator[StandardScalerModel]
- class StringIndexer(override val uid: String) extends Estimator[StringIndexerModel]
- class Tokenizer(override val uid: String) extends UnaryTransformer[String, Seq[String], Tokenizer]
- class RegexTokenizer(override val uid: String)
- class VectorAssembler(override val uid: String)
- class VectorIndexer(override val uid: String) extends Estimator[VectorIndexerModel]
- final class Word2Vec(override val uid: String) extends Estimator[Word2VecModel] with Word2VecBase
- class Param[T](val parent: String, val name: String, val doc: String, val isValid: T => Boolean)
- class DoubleParam(parent: String, name: String, doc: String, isValid: Double => Boolean)
- class IntParam(parent: String, name: String, doc: String, isValid: Int => Boolean)
- class FloatParam(parent: String, name: String, doc: String, isValid: Float => Boolean)
- class LongParam(parent: String, name: String, doc: String, isValid: Long => Boolean)
- class BooleanParam(parent: String, name: String, doc: String) // No need for isValid
- class ALS(override val uid: String) extends Estimator[ALSModel] with ALSParams
- final class DecisionTreeRegressor(override val uid: String)
- final class GBTRegressor(override val uid: String)
- class LinearRegression(override val uid: String)
- final class RandomForestRegressor(override val uid: String)
- class CrossValidator(override val uid: String) extends Estimator[CrossValidatorModel]
- trait Identifiable

AmplabJenkins · 2015-05-09T02:12:32Z

Build finished. Test FAILed.

AmplabJenkins · 2015-05-09T02:12:32Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32285/
Test FAILed.

AmplabJenkins · 2015-05-09T17:57:12Z

Merged build triggered.

AmplabJenkins · 2015-05-09T17:57:20Z

Merged build started.

SparkQA · 2015-05-09T17:59:08Z

Test build #32306 has started for PR 6019 at commit 629d402.

SparkQA · 2015-05-09T19:43:08Z

Test build #32306 has finished for PR 6019 at commit 629d402.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class Pipeline(override val uid: String) extends Estimator[PipelineModel]
- final class DecisionTreeClassifier(override val uid: String)
- final class GBTClassifier(override val uid: String)
- class LogisticRegression(override val uid: String)
- final class RandomForestClassifier(override val uid: String)
- class BinaryClassificationEvaluator(override val uid: String)
- final class Binarizer(override val uid: String)
- class ElementwiseProduct(override val uid: String)
- class HashingTF(override val uid: String) extends UnaryTransformer[Iterable[_], Vector, HashingTF]
- final class IDF(override val uid: String) extends Estimator[IDFModel] with IDFBase
- class Normalizer(override val uid: String) extends UnaryTransformer[Vector, Vector, Normalizer]
- class OneHotEncoder(override val uid: String)
- class PolynomialExpansion(override val uid: String)
- class StandardScaler(override val uid: String) extends Estimator[StandardScalerModel]
- class StringIndexer(override val uid: String) extends Estimator[StringIndexerModel]
- class Tokenizer(override val uid: String) extends UnaryTransformer[String, Seq[String], Tokenizer]
- class RegexTokenizer(override val uid: String)
- class VectorAssembler(override val uid: String)
- class VectorIndexer(override val uid: String) extends Estimator[VectorIndexerModel]
- final class Word2Vec(override val uid: String) extends Estimator[Word2VecModel] with Word2VecBase
- class Param[T](val parent: String, val name: String, val doc: String, val isValid: T => Boolean)
- class DoubleParam(parent: String, name: String, doc: String, isValid: Double => Boolean)
- class IntParam(parent: String, name: String, doc: String, isValid: Int => Boolean)
- class FloatParam(parent: String, name: String, doc: String, isValid: Float => Boolean)
- class LongParam(parent: String, name: String, doc: String, isValid: Long => Boolean)
- class BooleanParam(parent: String, name: String, doc: String) // No need for isValid
- class ALS(override val uid: String) extends Estimator[ALSModel] with ALSParams
- final class DecisionTreeRegressor(override val uid: String)
- final class GBTRegressor(override val uid: String)
- class LinearRegression(override val uid: String)
- final class RandomForestRegressor(override val uid: String)
- class CrossValidator(override val uid: String) extends Estimator[CrossValidatorModel]
- trait Identifiable

AmplabJenkins · 2015-05-09T19:43:13Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-09T19:43:13Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32306/
Test PASSed.

AmplabJenkins · 2015-05-13T07:12:10Z

Merged build triggered.

AmplabJenkins · 2015-05-13T07:12:18Z

Merged build started.

SparkQA · 2015-05-13T07:14:32Z

Test build #32594 has started for PR 6019 at commit 2569168.

SparkQA · 2015-05-13T08:56:18Z

Test build #32594 has finished for PR 6019 at commit 2569168.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class Pipeline(override val uid: String) extends Estimator[PipelineModel]
- final class DecisionTreeClassifier(override val uid: String)
- final class GBTClassifier(override val uid: String)
- class LogisticRegression(override val uid: String)
- final class OneVsRest(override val uid: String)
- final class RandomForestClassifier(override val uid: String)
- class BinaryClassificationEvaluator(override val uid: String)
- final class Binarizer(override val uid: String)
- final class Bucketizer(override val uid: String)
- class ElementwiseProduct(override val uid: String)
- class HashingTF(override val uid: String) extends UnaryTransformer[Iterable[_], Vector, HashingTF]
- final class IDF(override val uid: String) extends Estimator[IDFModel] with IDFBase
- class Normalizer(override val uid: String) extends UnaryTransformer[Vector, Vector, Normalizer]
- class OneHotEncoder(override val uid: String)
- class PolynomialExpansion(override val uid: String)
- class StandardScaler(override val uid: String) extends Estimator[StandardScalerModel]
- class StringIndexer(override val uid: String) extends Estimator[StringIndexerModel]
- class Tokenizer(override val uid: String) extends UnaryTransformer[String, Seq[String], Tokenizer]
- class RegexTokenizer(override val uid: String)
- class VectorAssembler(override val uid: String)
- class VectorIndexer(override val uid: String) extends Estimator[VectorIndexerModel]
- final class Word2Vec(override val uid: String) extends Estimator[Word2VecModel] with Word2VecBase
- class Param[T](val parent: String, val name: String, val doc: String, val isValid: T => Boolean)
- class DoubleParam(parent: String, name: String, doc: String, isValid: Double => Boolean)
- class IntParam(parent: String, name: String, doc: String, isValid: Int => Boolean)
- class FloatParam(parent: String, name: String, doc: String, isValid: Float => Boolean)
- class LongParam(parent: String, name: String, doc: String, isValid: Long => Boolean)
- class BooleanParam(parent: String, name: String, doc: String) // No need for isValid
- class ALS(override val uid: String) extends Estimator[ALSModel] with ALSParams
- final class DecisionTreeRegressor(override val uid: String)
- final class GBTRegressor(override val uid: String)
- class LinearRegression(override val uid: String)
- final class RandomForestRegressor(override val uid: String)
- class CrossValidator(override val uid: String) extends Estimator[CrossValidatorModel]
- trait Identifiable

AmplabJenkins · 2015-05-13T08:56:23Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-13T08:56:23Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32594/
Test PASSed.

jkbradley · 2015-05-13T18:34:35Z

examples/src/main/scala/org/apache/spark/examples/ml/DeveloperApiExample.scala

@@ -17,6 +17,7 @@

 package org.apache.spark.examples.ml

+import org.apache.spark.ml.util.Identifiable


Organize imports

jkbradley · 2015-05-13T18:37:55Z

@mengxr A bunch of small comments, but other than those, it looks good. At some point, it might be nice to add a test suite helper method which does standard checks to make sure models are handled correctly. E.g.: take (Estimator, Model, paramMap passed to fit()), and check UID, parent, and parameter values copied to the model. I just made a JIRA for that: [https://issues.apache.org/jira/browse/SPARK-7609]

AmplabJenkins · 2015-05-14T06:12:11Z

Merged build triggered.

AmplabJenkins · 2015-05-14T06:12:18Z

Merged build started.

SparkQA · 2015-05-14T06:14:16Z

Test build #32681 has started for PR 6019 at commit c4c8120.

SparkQA · 2015-05-14T07:59:57Z

Test build #32681 has finished for PR 6019 at commit c4c8120.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-14T08:00:01Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-14T08:00:02Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32681/
Test PASSed.

A param instance is strongly attached to an parent in the current implementation. So if we make a copy of an estimator or a transformer in pipelines and other meta-algorithms, it becomes error-prone to copy the params to the copied instances. In this PR, a param is identified by its parent's UID and the param name. So it becomes loosely attached to its parent and all its derivatives. The UID is preserved during copying or fitting. All components now have a default constructor and a constructor that takes a UID as input. I keep the constructors for Param in this PR to reduce the amount of diff and moved `parent` as a mutable field. This PR still needs some clean-ups, and there are several spark.ml PRs pending. I'll try to get them merged first and then update this PR. jkbradley Author: Xiangrui Meng <[email protected]> Closes #6019 from mengxr/SPARK-7407 and squashes the following commits: c4c8120 [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into SPARK-7407 520f0a2 [Xiangrui Meng] address comments 2569168 [Xiangrui Meng] fix tests 873caca [Xiangrui Meng] fix tests in OneVsRest; fix a racing condition in shouldOwn 409ea08 [Xiangrui Meng] minor updates 83a163c [Xiangrui Meng] update JavaDeveloperApiExample 5db5325 [Xiangrui Meng] update OneVsRest 7bde7ae [Xiangrui Meng] merge master 697fdf9 [Xiangrui Meng] update Bucketizer 7b4f6c2 [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into SPARK-7407 629d402 [Xiangrui Meng] fix LRSuite 154516f [Xiangrui Meng] merge master aa4a611 [Xiangrui Meng] fix examples/compile a4794dd [Xiangrui Meng] change Param to use to reduce the size of diff fdbc415 [Xiangrui Meng] all tests passed c255f17 [Xiangrui Meng] fix tests in ParamsSuite 818e1db [Xiangrui Meng] merge master e1160cf [Xiangrui Meng] fix tests fbc39f0 [Xiangrui Meng] pass test:compile 108937e [Xiangrui Meng] pass compile 8726d39 [Xiangrui Meng] use parent uid in Param eaeed35 [Xiangrui Meng] update Identifiable (cherry picked from commit 1b8625f) Signed-off-by: Xiangrui Meng <[email protected]>

mengxr · 2015-05-14T08:24:27Z

Merged into master and branch-1.4. I will send another PR for SPARK-7609.

A param instance is strongly attached to an parent in the current implementation. So if we make a copy of an estimator or a transformer in pipelines and other meta-algorithms, it becomes error-prone to copy the params to the copied instances. In this PR, a param is identified by its parent's UID and the param name. So it becomes loosely attached to its parent and all its derivatives. The UID is preserved during copying or fitting. All components now have a default constructor and a constructor that takes a UID as input. I keep the constructors for Param in this PR to reduce the amount of diff and moved `parent` as a mutable field. This PR still needs some clean-ups, and there are several spark.ml PRs pending. I'll try to get them merged first and then update this PR. jkbradley Author: Xiangrui Meng <[email protected]> Closes apache#6019 from mengxr/SPARK-7407 and squashes the following commits: c4c8120 [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into SPARK-7407 520f0a2 [Xiangrui Meng] address comments 2569168 [Xiangrui Meng] fix tests 873caca [Xiangrui Meng] fix tests in OneVsRest; fix a racing condition in shouldOwn 409ea08 [Xiangrui Meng] minor updates 83a163c [Xiangrui Meng] update JavaDeveloperApiExample 5db5325 [Xiangrui Meng] update OneVsRest 7bde7ae [Xiangrui Meng] merge master 697fdf9 [Xiangrui Meng] update Bucketizer 7b4f6c2 [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into SPARK-7407 629d402 [Xiangrui Meng] fix LRSuite 154516f [Xiangrui Meng] merge master aa4a611 [Xiangrui Meng] fix examples/compile a4794dd [Xiangrui Meng] change Param to use to reduce the size of diff fdbc415 [Xiangrui Meng] all tests passed c255f17 [Xiangrui Meng] fix tests in ParamsSuite 818e1db [Xiangrui Meng] merge master e1160cf [Xiangrui Meng] fix tests fbc39f0 [Xiangrui Meng] pass test:compile 108937e [Xiangrui Meng] pass compile 8726d39 [Xiangrui Meng] use parent uid in Param eaeed35 [Xiangrui Meng] update Identifiable

mengxr added 9 commits May 6, 2015 21:08

update Identifiable

eaeed35

use parent uid in Param

8726d39

pass compile

108937e

pass test:compile

fbc39f0

fix tests

e1160cf

merge master

818e1db

fix tests in ParamsSuite

c255f17

all tests passed

fdbc415

change Param to use to reduce the size of diff

a4794dd

fix examples/compile

aa4a611

mengxr added 2 commits May 8, 2015 23:59

merge master

154516f

fix LRSuite

629d402

jkbradley reviewed May 13, 2015
View reviewed changes

mengxr added 2 commits May 13, 2015 22:56

address comments

520f0a2

Merge remote-tracking branch 'apache/master' into SPARK-7407

c4c8120

asfgit closed this in 1b8625f May 14, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-7407][MLLIB] use uid + name to identify parameters #6019

[SPARK-7407][MLLIB] use uid + name to identify parameters #6019

mengxr commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

SparkQA commented May 8, 2015

SparkQA commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 9, 2015

SparkQA commented May 9, 2015

SparkQA commented May 9, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 9, 2015

SparkQA commented May 9, 2015

SparkQA commented May 9, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 13, 2015

AmplabJenkins commented May 13, 2015

SparkQA commented May 13, 2015

SparkQA commented May 13, 2015

AmplabJenkins commented May 13, 2015

AmplabJenkins commented May 13, 2015

jkbradley May 13, 2015

mengxr May 14, 2015

jkbradley commented May 13, 2015

AmplabJenkins commented May 14, 2015

AmplabJenkins commented May 14, 2015

SparkQA commented May 14, 2015

SparkQA commented May 14, 2015

AmplabJenkins commented May 14, 2015

AmplabJenkins commented May 14, 2015

mengxr commented May 14, 2015

		@@ -17,6 +17,7 @@

		package org.apache.spark.examples.ml

		import org.apache.spark.ml.util.Identifiable

[SPARK-7407][MLLIB] use uid + name to identify parameters #6019

[SPARK-7407][MLLIB] use uid + name to identify parameters #6019

Conversation

mengxr commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

SparkQA commented May 8, 2015

SparkQA commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 9, 2015

SparkQA commented May 9, 2015

SparkQA commented May 9, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 9, 2015

SparkQA commented May 9, 2015

SparkQA commented May 9, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 13, 2015

AmplabJenkins commented May 13, 2015

SparkQA commented May 13, 2015

SparkQA commented May 13, 2015

AmplabJenkins commented May 13, 2015

AmplabJenkins commented May 13, 2015

jkbradley May 13, 2015

Choose a reason for hiding this comment

mengxr May 14, 2015

Choose a reason for hiding this comment

jkbradley commented May 13, 2015

AmplabJenkins commented May 14, 2015

AmplabJenkins commented May 14, 2015

SparkQA commented May 14, 2015

SparkQA commented May 14, 2015

AmplabJenkins commented May 14, 2015

AmplabJenkins commented May 14, 2015

mengxr commented May 14, 2015