v.2.4.8

• Bug fixed:
  • Wrong table schema in Union expression when pushdown
• Drop Spark 2.1 support since this version
• Support pre-release Spark 2.4 for test only

v.2.4.7

Bug fixed:

• Errors in loading from query #79

v.2.4.6

fixed bugs:

• always drop tables after renamed staging tables
• column name mismatch in pushdown function

v.2.4.5

Bug Fixed:

• Can not insert Timestamp value into Date column.

v.2.4.4

Support variant data types(MapType, StructType, and ArrayType)

• Write to Snowflake
  • Spark-Snowflake Connector can correctly write the variant data type to Snowflake table now. (In previous versions, all variant data types would be converted to StringType before written to Snowflake table.)
  • The staging data file format will be JSON instead of CSV, if the Spark Dataframe contains variant data types.
  • Example:
    Spark Dataframe Schema

StructType(
    Array(
      StructField("ARR", ArrayType(IntegerType), false),
      StructField("OB",StructType(
        Array(
          StructField("str", StringType, false)
        )
      ), false),
      StructField("MAP", MapType(StringType,StringType), false),
      StructField("NUM", IntegerType, false)
    )
  )

Result Snowflake Table Schema

(ARR VARIANT, OB VARIANT, MAP VARIANT, NUM INTEGER)

• Read from Snowflake
  • If a schema containing variant data types provided during reading from Snowflake, the connector will convert all variant data to the corresponding types.
  • Otherwise, variant data will be converted to StringType by default.
  • Example:
    Snowflake Table Schema

(NUM INTEGER, STR STRING, ARR VARIANT)

Read From Snowflake

  val schema = new StructType(
    Array(
      StructField("NUM", IntegerType, false),
      StructField("STR", StringType, false),
      StructField("ARR", ArrayType(IntegerType), false)
  )

  val df = sqlContext.read
      .format(SNOWFLAKE_SOURCE_NAME)
      .options(sfOption)
      .schema(schema)
      .load()

v.2.4.3

• Support Column Mapping function on external stage
• Bug fixed:
  • No default value of truncate_table variable in v2.4.2

v.2.4.2

• Add new parameter truncate_table
  • Value: on/off
  • Default: off
  • Detail: If it is on, the schema of original table will be kept in overwrite mode when writing data.
• Add new parameter continue_on_error
  • Value: on/off
  • Default: off
  • Detail: if it is on, the on_error parameter will be continue in COPY command.

v.2.4.1

• Fixed an issue when including line feed in string values
• Fixed an issue when using Azure external stages in IT test

v.2.4.0

• Support Azure internal and external stage

v.2.3.3

• Introduced purge parameter to enable/disable automatic purging external storage when writing data to Snowflake.