Skip to content

Commit

Permalink
[SPARK-18167] Re-enable the non-flaky parts of SQLQuerySuite
Browse files Browse the repository at this point in the history
## What changes were proposed in this pull request?

It seems the proximate cause of the test failures is that `cast(str as decimal)` in derby will raise an exception instead of returning NULL. This is a problem since Hive sometimes inserts `__HIVE_DEFAULT_PARTITION__` entries into the partition table as documented here: https://github.com/apache/hive/blob/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L1034

Basically, when these special default partitions are present, partition pruning pushdown using the SQL-direct mode will fail due this cast exception. As commented on in `MetaStoreDirectSql.java` above, this is normally fine since Hive falls back to JDO pruning, however when the pruning predicate contains an unsupported operator such as `>`, that will fail as well.

The only remaining question is why this behavior is nondeterministic. We know that when the test flakes, retries do not help, therefore the cause must be environmental. The current best hypothesis is that some config is different between different jenkins runs, which is why this PR prints out the Spark SQL and Hive confs for the test. The hope is that by comparing the config state for failure vs success we can isolate the root cause of the flakiness.

**Update:** we could not isolate the issue. It does not seem to be due to configuration differences. As such, I'm going to enable the non-flaky parts of the test since we are fairly confident these issues only occur with Derby (which is not used in production).

## How was this patch tested?

N/A

Author: Eric Liang <[email protected]>

Closes #15725 from ericl/print-confs-out.
  • Loading branch information
ericl authored and yhuai committed Nov 4, 2016
1 parent 550cd56 commit 4cee2ce
Showing 1 changed file with 10 additions and 21 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -1569,27 +1569,16 @@ class SQLQuerySuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
).map(i => Row(i._1, i._2, i._3, i._4)))
}

ignore("SPARK-10562: partition by column with mixed case name") {
def runOnce() {
withTable("tbl10562") {
val df = Seq(2012 -> "a").toDF("Year", "val")
df.write.partitionBy("Year").saveAsTable("tbl10562")
checkAnswer(sql("SELECT year FROM tbl10562"), Row(2012))
checkAnswer(sql("SELECT Year FROM tbl10562"), Row(2012))
checkAnswer(sql("SELECT yEAr FROM tbl10562"), Row(2012))
checkAnswer(sql("SELECT val FROM tbl10562 WHERE Year > 2015"), Nil)
checkAnswer(sql("SELECT val FROM tbl10562 WHERE Year == 2012"), Row("a"))
}
}
try {
runOnce()
} catch {
case t: Throwable =>
// Retry to gather more test data. TODO(ekl) revert this once we deflake this test.
runOnce()
runOnce()
runOnce()
throw t
test("SPARK-10562: partition by column with mixed case name") {
withTable("tbl10562") {
val df = Seq(2012 -> "a").toDF("Year", "val")
df.write.partitionBy("Year").saveAsTable("tbl10562")
checkAnswer(sql("SELECT year FROM tbl10562"), Row(2012))
checkAnswer(sql("SELECT Year FROM tbl10562"), Row(2012))
checkAnswer(sql("SELECT yEAr FROM tbl10562"), Row(2012))
// TODO(ekl) this is causing test flakes [SPARK-18167], but we think the issue is derby specific
// checkAnswer(sql("SELECT val FROM tbl10562 WHERE Year > 2015"), Nil)
checkAnswer(sql("SELECT val FROM tbl10562 WHERE Year == 2012"), Row("a"))
}
}

Expand Down

0 comments on commit 4cee2ce

Please sign in to comment.