The current version of the migrator is not compatible with the Spark version on EMR

[julienrf]

Currently, the migrator uses Scala 2.13 whereas EMR provides out-of-the-box support for Spark on Scala 2.12. This makes it impossible to use the Migrator on EMR using the built-in Spark images.

There two possible solutions:

1. Switch to Scala 2.12 in the migrator. This should be simple to do (maybe we have to add a dependency on scala-collection-compat).
2. Document how to build custom EMR images, or provide our own EMR images correctly pre-configured. This option needs to be investigated.

[lukasbraach]

@julienrf yes, I agree. I am also struggling with the same problem.

Building a custom image will not easily be possible, since Amazon EMR uses a custom version of Apache Spark, and therefore we cannot custom-compile it with another Scala version. So the only option for EMR is to ensure compatibility with Scala 2.12.

If this is not easily fixable, we should at least remove or adapt the documentation section around EMR.