[SPARK-3292][streaming]Fix:Shuffle Tasks run incessantly even though there's no inputs #2192
Conversation
|
Can one of the admins verify this patch? |
|
i recommit this.for the last PR is compicated by mistake |
|
Can one of the admins verify this patch? |
|
@tdas can you take a look at this? |
|
This is not a good idea. Not returning an RDD can mess up a lot of the logic and semantics. For example if there is a transform() followed by updateStateByKey(), the result will be unpredictable. updateStateByKey expects the previous batch to have a state RDD. If it does not find any state RDD it will assume that this the start of the streamign computation and effectively initialize again, forgetting the previous states from 2 batches ago. So this change is incorrect. |
|
@guowei2 As i had explained, this is not a good idea because it breaks semantics for a state dstream. Mind closing this PR? |
with this PR: no job is commited when there's no imputs.