Skip to content

Commit

Permalink
Adds comment
Browse files Browse the repository at this point in the history
  • Loading branch information
liancheng committed Feb 25, 2016
1 parent 51ad500 commit 24b96bd
Showing 1 changed file with 5 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -804,6 +804,11 @@ object SimplifyFilters extends Rule[LogicalPlan] {
*/
object PushPredicateThroughProject extends Rule[LogicalPlan] with PredicateHelper {
def apply(plan: LogicalPlan): LogicalPlan = plan transform {
// SPARK-13473: We can't push the predicate down when the underlying projection output non-
// deterministic field(s). Non-deterministic expressions are essentially stateful. This
// implies that, for a given input row, the output are determined by the expression's initial
// state and all the input rows processed before. In another word, the order of input rows
// matters for non-deterministic expressions, while pushing down predicates changes the order.
case filter @ Filter(condition, project @ Project(fields, grandChild))
if fields.forall(_.deterministic) =>

Expand Down

0 comments on commit 24b96bd

Please sign in to comment.