[SPARK-6963][CORE]Flaky test: o.a.s.ContextCleanerSuite automatically cleanup checkpoint

[witgo]

cc @andrewor14

[SparkQA]

Test build #30453 has finished for PR 5548 at commit b08b3c9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[srowen]

I'd like to start pushing back on PRs without explanation, here or in the JIRA. Please add a short explanation of why this is the right change and what the problem is. Otherwise I think we need to start rejecting these.

[witgo]

@srowen This is just a test code bug. new CleanerTester(sc, Seq(rddId), Nil, Nil) This code is only to ensure that the RDD is cleaned, but checkpoint. rdd and checkpoint almost simultaneously be cleaned, But there are exceptions, depending on the GC. The checkpointCleaned ensure the checkpoint is cleaned.

[srowen]

How does this fix the assertion that failed? you add a new set of IDs that must now be counted down by a new callback, and you assert it ends up empty, which is good. But the assertion that failed was earlier:

     assert(fs.exists(RDDCheckpointData.rddCheckpointDataPath(sc, rddId).get))

[witgo]

The tips are more confusing.
The jira display the error in: ContextCleanerSuite.scala#L252.
The code in #L252 is assert(!fs.exists(RDDCheckpointData.rddCheckpointDataPath(sc, rddId).get))

[srowen]

Ah right, it is the following one. Still, how does this affect whether the checkpoint files exist?

[SparkQA]

Test build #30476 has finished for PR 5548 at commit 964aea7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.