[luigi.contrib.pyspark_runner] SparkSession support in PySparkTask

[drowoseque]

Description

luigi.contrib.spark.PySparkTask now supports spark_session as a first argument in main().
To use it just enable pyspark_runner.use_spark_session = True in luigi.cfg
This change allows users to work with spark_sesion it in their jobs on Apache Spark 2+ version.

Motivation and Context

Apache Spark 2.0.0 have introduced a new entrypoint object which is SparkSession instead of SparkContext in previous releases.
luigi.contrib.spark.PySparkTask in its main function supports only sc argument which doesn't allow you to use spark session object.

Have you tested this? If so, how?

• I've added unit tests into my PR
• I've checked locally that this works as expected (for both old-style and new-style syntax)

[drowoseque]

@dlstadther
@honnix
@ntim
please, take a look at that

[Tarrasch]

You probably want other Spark users to comment on the usefulness of this PR. :)

[drowoseque]

You probably want other Spark users to comment on the usefulness of this PR. :)

yes, but none of them (PysparkTask contributors) seem to be online last year except @ntim and you @Tarrasch

[Tarrasch]

I didn't check the details. But I'm fine with merge.

Can you get a coworker to read this code too?

[drowoseque]

@b2arn
PTAL

[drowoseque]

@Tarrasch done

[drowoseque]

@honnix
@dlstadther
PTAL

[dlstadther]

lgtm

[drowoseque]

@GoodDok
@mrk-its
PTAL as well

[drowoseque]

@Tarrasch @honnix
please up