What is the reason this project uses a different Spark version for testing?

[paulsasi]

Hi!

I've noticed the project's support for Spark 3.0.0, yet the build.sbt file employs Spark 2.4.5 for testing:

val sparkVersion = "3.0.0"
val sparkTestVersion = "2.4.5"
...
"com.holdenkarau" %% "spark-testing-base" % s"${sparkTestVersion}_0.14.0" % Test

I'm curious about the rationale behind this. Wouldn't it be more suitable to align the testing version with 3.0.0?

Attempting an upgrade, I found that there's no equivalent to com.holdenkarau.spark.testing.HDFSCluster in spark-testing-base version 1.X.X, a class heavily relied upon in our tests. (holdenk/spark-testing-base@09b031a)

Considering this, do you think it's worthwhile to invest effort in adapting the tests to accommodate Spark 3.0.0?