Skip to content

Commit

Permalink
[Ml] SPARK-5796 Don't transform data on a last estimator in Pipeline
Browse files Browse the repository at this point in the history
If it's a last estimator in Pipeline there's no need to transform data, since there's no next stage that would consume this data.
  • Loading branch information
petro-rudenko committed Feb 13, 2015
1 parent 1768bd5 commit d13ec33
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,9 @@ class Pipeline extends Estimator[PipelineModel] {
throw new IllegalArgumentException(
s"Do not support stage $stage of type ${stage.getClass}")
}
curDataset = transformer.transform(curDataset, paramMap)
if (index < indexOfLastEstimator) {
curDataset = transformer.transform(curDataset, paramMap)
}
transformers += transformer
} else {
transformers += stage.asInstanceOf[Transformer]
Expand Down

0 comments on commit d13ec33

Please sign in to comment.