[Improvement] [spark-connector] add IntervalYearType and IntervalDayType for spark connector

[FANNG1]

What would you like to be improved?

you could refer #2484 add IntervalYearType and IntervalDayType support for spark connector, add transform logic in SparkTypeConverter and add test in TestSparkTypeConverter

How should we improve?

No response

[FANNG1]

@mchades , I checked spark types, Gravitino IntervalYearType and IntervalDayType couldn't convert to Spark YearMonthIntervalType(startField, endField) and DayTimeIntervalType(startField, endField) because they have parameters. Yes?

[mchades]

Gravitino IntervalYearType can convert to Spark YearMonthIntervalType(YEAR, MONTH)
Gravitino IntervalDayType can convert to Spark DayTimeIntervalType(DAY, SECOND)

FYI: https://substrait.io/types/type_classes/

[charliecheng630]

@FANNG1 @mchades I would like to work on this.
After tracing code, should IntervalYearType and IntervalDayType be added with parameters (startField, endField)?

[FANNG1]

@FANNG1 @mchades I would like to work on this. After tracing code, should IntervalYearType and IntervalDayType be added with parameters (startField, endField)?

I'm not sure about this, How should Spark DayTimeIntervalType(DAY, MINUTE) translate to Gravitino? @mchades

[charliecheng630]

@mchades Do we need to add parameters in Types? If needed, I would like to implement.

[mchades]

I think we should understand how the interval type is actually used in Spark in order to determine if additional parameters are needed.
Could you provide code examples of how Spark defines and reads/writes the interval type? @FANNG1

[FANNG1]

Take YearMonthIntervalType for example, it has two parameters startField and endFiled which could be YEAR and MONTH, valid types are:

• interval year, like INTERVAL '100' YEAR
• interval month, like INTERVAL '2' MONTH
• interval year to month, like INTERVAL '100 2' YEAR TO MONTH

  override val typeName: String = {
    val startFieldName = fieldToString(startField)
    val endFieldName = fieldToString(endField)
    if (startFieldName == endFieldName) {
      s"interval $startFieldName"
    } else if (startField < endField) {
      s"interval $startFieldName to $endFieldName"
    } else {
      throw QueryCompilationErrors.invalidDayTimeIntervalType(startFieldName, endFieldName)
    }
  }

[mchades]

This issue should be considered together with #2373

[FANNG1]

Gravitino IntervalYearType can convert to Spark YearMonthIntervalType(YEAR, MONTH) Gravitino IntervalDayType can convert to Spark DayTimeIntervalType(DAY, SECOND)

FYI: https://substrait.io/types/type_classes/

seems we could do the transform like this, because the simpleString of YearMonthIntervalType is YearMonthIntervalType(YEAR, MONTH) which is used to transform to hive types in spark. @mchades WDYT?

[mchades]

Gravitino IntervalYearType can convert to Spark YearMonthIntervalType(YEAR, MONTH) Gravitino IntervalDayType can convert to Spark DayTimeIntervalType(DAY, SECOND)
FYI: https://substrait.io/types/type_classes/

seems we could do the transform like this, because the simpleString of YearMonthIntervalType is YearMonthIntervalType(YEAR, MONTH) which is used to transform to hive types in spark. @mchades WDYT?

we can convert it like this first.

[FANNG1]

@charliecheng630 do you still like to work on this?

[FANNG1]

@charliecheng630 , this issue is expected to be fix in 0.5 in next week, If you didn't have time, I'd like to work on this.

[charliecheng630]

@FANNG1 I probably won't make it in time to work on this issue.

[FANNG1]

Spark don't support create table or load table with IntervalXXType column, not plan to support it unless a real need.