[SPARK-4856] [SQL] NullType instead of StringType when sampling against empty string or nul...

[chenghao-intel]

TestSQLContext.sparkContext.parallelize(
  """{"ip":"27.31.100.29","headers":{"Host":"1.abc.com","Charset":"UTF-8"}}""" ::
  """{"ip":"27.31.100.29","headers":{}}""" ::
  """{"ip":"27.31.100.29","headers":""}""" :: Nil)

As empty string (the "headers") will be considered as String in the beginning (in line 2 and 3), it ignores the real nested data type (struct type "headers" in line 1), and also take the line 1 (the "headers") as String Type, which is not our expected.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24479/
Test FAILed.

[chenghao-intel]

retest this please

[SparkQA]

Test build #24481 has started for PR 3708 at commit 853de51.

• This patch merges cleanly.

[SparkQA]

Test build #24481 has finished for PR 3708 at commit 853de51.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Boolean)

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24481/
Test PASSed.

[marmbrus]

The null case makes sense to me, but why "" as well? That seems to be unequivocally a String

[chenghao-intel]

I've updated the unit test, which probably make more sense.

In some cases (as shown in the unit test), "" is equivalent to null for struct type, so we'd better not to say "it's MUST be StringType if we meet an empty string".

In the meantime, the NullType is the minimum data type, and it can be promoted to any other data type in JsonRDD (e.g. promote to StructType), however, it's impossible to promote a StringType to StructType.

It's safe to make it as NullType here, as we can promote it as StringType in the last promote rules, see https://github.com/chenghao-intel/spark/blob/json/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala#L231

[SparkQA]

Test build #24519 has started for PR 3708 at commit e7a72e9.

• This patch merges cleanly.

[SparkQA]

Test build #24519 has finished for PR 3708 at commit e7a72e9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Boolean)

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24519/
Test PASSed.

[marmbrus]

Thanks! Merged to master.