[SPARK-4213][SQL][WIP] ParquetFilters - No support for LT, LTE, GT, GTE operators

[sarutak]

Following description is quoted from JIRA:

When I issue a hql query against a HiveContext where my predicate uses a column of string type with one of LT, LTE, GT, or GTE operator, I get the following error:
scala.MatchError: StringType (of class org.apache.spark.sql.catalyst.types.StringType$)
Looking at the code in org.apache.spark.sql.parquet.ParquetFilters, StringType is absent from the corresponding functions for creating these filters.
To reproduce, in a Hive 0.13.1 shell, I created the following table (at a specified DB):

create table sparkbug (
id int,
event string
) stored as parquet;

Insert some sample data:

insert into table sparkbug select 1, '2011-06-18' from <some table> limit 1;
insert into table sparkbug select 2, '2012-01-01' from <some table> limit 1;

Launch a spark shell and create a HiveContext to the metastore where the table above is located.

import org.apache.spark.sql._
import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.hive.HiveContext
val hc = new HiveContext(sc)
hc.setConf("spark.sql.shuffle.partitions", "10")
hc.setConf("spark.sql.hive.convertMetastoreParquet", "true")
hc.setConf("spark.sql.parquet.compression.codec", "snappy")
import hc._
hc.hql("select * from <db>.sparkbug where event >= '2011-12-01'")

A scala.MatchError will appear in the output.

[sarutak]

CC @marmbrus

[SparkQA]

Test build #22840 has finished for PR 3083 at commit 9a1fae7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class NullType(PrimitiveType):
  • case class ScalaUdfBuilder[T: TypeTag](f: AnyRef)

[marmbrus]

What about DateType and the others? Do we need a default case that avoids match errors when we have predicates on types that parquet can't handle pushdown for?

[sarutak]

Yeah, exactly. I'll try for the rest of the types.

[SparkQA]

Test build #22903 has finished for PR 3083 at commit 4ab6e56.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • final class ByteColumn(columnPath: ColumnPath)
  • final class ShortColumn(columnPath: ColumnPath)
  • final class DateColumn(columnPath: ColumnPath)
  • final class TimestampColumn(columnPath: ColumnPath)
  • final class DecimalColumn(columnPath: ColumnPath)
  • final class WrappedDate(val date: Date) extends Comparable[WrappedDate]
  • final class WrappedTimestamp(val timestamp: Timestamp) extends Comparable[WrappedTimestamp]

[marmbrus]

Thanks! Merged to master and 1.2

[sarutak]

@marmbrus Please wait. I found this change has still problem. I'll address it.

[marmbrus]

Sorry, too late. Please address in a follow up.

Can you put [WIP] in the title when its not ready to merge?