Skip to content

Commit

Permalink
[SPARK-48544][SQL] Reduce memory pressure of empty TreeNode BitSets
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

- Changed the `ineffectiveRules` variable of the `TreeNode` class to initialize lazily. This will reduce unnecessary driver memory pressure.

### Why are the changes needed?

- Plans with large expression or operator trees are known to cause driver memory pressure; this is one step in alleviating that issue.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Existing UT covers behavior. Outwards facing behavior does not change.

### Was this patch authored or co-authored using generative AI tooling?

No

Closes apache#46919 from n-young-db/ineffective-rules-lazy.

Authored-by: Nick Young <[email protected]>
Signed-off-by: Josh Rosen <[email protected]>
  • Loading branch information
n-young-db authored and JoshRosen committed Jun 10, 2024
1 parent ec6db63 commit 5a2f374
Showing 1 changed file with 9 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,14 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]]
* ineffective for subsequent apply calls on this tree because query plan structures are
* immutable.
*/
private val ineffectiveRules: BitSet = new BitSet(RuleIdCollection.NumRules)
private[this] var _ineffectiveRules: BitSet = null
private def ineffectiveRules: BitSet = {
if (_ineffectiveRules eq null) {
_ineffectiveRules = new BitSet(RuleIdCollection.NumRules)
}
_ineffectiveRules
}
private def isIneffectiveRulesEmpty = _ineffectiveRules eq null

/**
* @return a sequence of tree pattern enums in a TreeNode T. It does not include propagated
Expand Down Expand Up @@ -149,7 +156,7 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]]
* UnknownId, it returns false.
*/
protected def isRuleIneffective(ruleId : RuleId): Boolean = {
if (ruleId eq UnknownRuleId) {
if (isIneffectiveRulesEmpty || (ruleId eq UnknownRuleId)) {
return false
}
ineffectiveRules.get(ruleId.id)
Expand Down

0 comments on commit 5a2f374

Please sign in to comment.