Release Spark Connector 2.11.1

Added support for AWS VPCE deployments and fixed some bugs:

1. Support AWS VPCE. Added the configuration option S3_STAGE_VPCE_DNS_NAME for specifying the VPCE DNS name at the session level.
2. Updated the connector to close JDBC connections to avoid connection leakage.
3. Fixed a NullPointerException issue when sending telemetry messages.
4. Added a new configuration option treat_decimal_as_long to enable the Spark Connector to return Long values instead of BigDecimal values, if the query returns Decimal(<any_precision>, 0). WARNING: If the value is greater than the maximum value of Long, an error will be raised.
5. Added a new option proxy_protocol for specifying the proxy protocol (http or https) with AWS deployments. (The option has no effect on Azure and GCP deployments.)
6. Added support for counting rows in a table where the row count is greater than the maximum value of Integer.
7. Updated the connector to use the Snowflake JDBC driver 3.13.24.

Release Spark Connector 2.11.0

Support Spark 3.3 and fixed some bugs:

1. Added support for Spark 3.3.
2. Upgraded the version of the PostgreSQL JDBC Driver that tests depend on to avoid the security vulnerability CVE-2022-31197.
3. Updated the connector to use the Snowflake JDBC driver 3.13.22 and the Snowflake Ingest SDK 0.10.8.

NOTE:

1. Starting from version 2.11.0, the Snowflake Connector for Spark supports Spark 3.1, 3.2 and 3.3.
   Version 2.11.0 of the Snowflake Connector for Spark does not support Spark 3.0. Note that previous versions of the connector continue to support Spark 3.0.
2. For Snowflake GCP accounts, the Snowflake JDBC driver versions 3.13.16 through 3.13.21 do not work with the Spark connector.

Release Spark Connector 2.10.1

Fixed some critical issues:

1. Removed unnecessary dependencies on libraries to avoid the security vulnerabilities CVE-2020-8908 and CVE-2018-10237.
2. Added support for using the JDBC data type TIMESTAMP_WITH_TIMEZONE when reading data from Snowflake.
3. Changed the logic for checking for the existence of a table before saving a DataFrame to Snowflake:
   • The connector now reuses the existing connection (rather than creating a new connection) to avoid potential problems with token expiration.
   • If the table name is not fully qualified (i.e. does not include the schema name), the connector now checks for the table under the schema specified by sfSchema, rather than the schema that is currently in use in the session.
     Note: If you need to save a DataFrame to a table in a schema other than sfSchema, specify the schema as part of the fully qualified name of the table, rather than executing USE SCHEMA to change the current schema.
4. Improved performance by avoiding unnecessary parse_json() calls in the COPY INTO TABLE command when writing a DataFrame with ArrayType, MapType or StructType columns to Snowflake.
5. Added the getLastSelectQueryId and getLastCopyLoadQueryId methods to the Utils class. These methods return the query ID of the last query that read data from Snowflake and the last COPY INTO TABLE statement that was executed (respectively).

Release Spark Connector 2.10.0

• Support Spark 3.2
• Fix an issue that string "null" is regarded as NULL when pushding

Note:

1. From this version, Spark connector will support spark 3.0, 3.1 and 3.2, Spark 2.4 is not supported any more

Release Spark Connector 2.9.3

Fixed some critical issues:

1. Modified the connector to avoid executing a CREATE TABLE command when writing an empty DataFrame to a Snowflake table if the target table exists and the following options are set for the connector: “usestagingtable” is set to “off” and “truncate_table” is set to “on”.
2. Added support for writing a DataFrame to a Snowflake table when the Snowflake account has FORCE_REGIONAL_S3_ENDPOINTS_FOR_PRESIGNED_URLS enabled.
3. Fixed a potential memory leak that could occur when a DataFrame reads from Snowflake tables but the results are not consumed completely.
4. Updated the connector to use the latest version of the Snowflake JDBC Driver (3.13.14).

Release Spark Connector 2.9.2

Fixed some critical issues:

1. For the DataFrame read and write queries, the connector now logs the query ID and the URL to display the query details.
2. When a Spark application fails or is cancelled, the connector now cancels all running queries executed by that Spark application.
3. Added the configuration options timestamp_ntz_output_format, timestamp_ltz_output_format, and timestamp_tz_output_format for specifying the output format for timestamp values. By default, these are set as "YYYY-MM-DD HH24:MI:SS.FF3", "TZHTZM YYYY-MM-DD HH24:MI:SS.FF3", and "TZHTZM YYYY-MM-DD HH24:MI:SS.FF3". If they are set to “sf_current”, which means that the connector uses the formats specified for the session.
4. Updated the connector to use the latest version of the Snowflake JDBC Driver (3.13.10).

Release Spark Connector 2.9.1

Fixed some critical issues:

1. Improved the way in which the connector retrieves metadata for a schema or query. In the past, the connector executed a query to get the metadata. With this release, the connector uses a JDBC PreparedStatement to retrieve just the ResultSet metadata.
2. Fixed an issue with pushdowns when you call limit() after sort() in a DataFrame transformation.
3. Updated the connector to use the latest version of the Snowflake JDBC Driver (3.13.6).
4. Updated the connector to use the latest version of the Snowflake Ingest SDK (0.10.3).

Release Spark Connector 2.9.0

Support spark 3.1 and some bug fixes.

1. Added support for Spark 3.1.
2. Removed the requirement that you must have the privilege to set and get system properties.
3. Updated the connector to use the latest version of the Snowflake JDBC Driver (3.13.3).
4. Updated the connector to use the latest version of the Snowflake Ingest SDK (0.10.2).

Note:

1. With this new version of the connector, Snowflake continues to support the latest three Spark versions: Spark 2.4, 3.0 and 3.1.
2. Version 2.9.0 of the Snowflake Connector for Spark does not support Spark 2.3. Note that previous versions of the connector continue to support Spark 2.3.
3. Spark Connector 2.9.0 is NOT compatible with version 3.13.2 (and earlier versions) of the JDBC Driver.

Release Spark Connector 2.8.6

This release is used to confirm the new spark connector release process to work. It's functionality is equal to Spark Connector 2.8.5.

Release Spark Connector 2.8.5

Fixed some critical issues:

1. Added pushdown support for the EqualNullSafe operator.
2. In cases where Date column data is converted to Number column data (e.g. an integer or long), avoid performing a pushdown, which would result in conversion errors.
3. In cases where DataFrame.dropDuplicates() is called with only some of the columns in the DataFrame (e.g. DataFrame.dropDuplicates("C1") on a DataFrame with the columns "C1" and "C2"), avoid performing a pushdown, which would result in a syntax error.
4. Added support to write a DataFrame that has no partitions to the Snowflake database. If the target table doesn’t exist, an empty table is created. If the target table exists and the write mode is "overwrite", the target table is overwritten with an empty table.
5. Disabled the pushdown of some window functions in order to avoid inconsistent results.
6. Fixed an issue with writing a DataFrame to a Snowflake table when the DataFrame has a column with a complex type (e.g. MapType) and the column name is Snowflake SQL keyword.
7. Updated the connector to use the latest version of the Snowflake JDBC Driver (3.13.2).