Timezone support #5

stheppi · 2024-02-23T12:23:52Z

There have been requests from users to support date/time partition in the S3/GCP/Azure sink with a specific timezone. The code changes brings the timezone configuration as an optional setting. When epoch configuration is involved, changing the timezone to non-UTC yields an error.

Removes the check plugin since the jar dependency is built with a newer JVM version and fails the build.

There have been requests from users to support date/time partition in the S3/GCP/Azure sink with a specific timezone. The code changes brings the timezone configuration as an optional setting. When epoch configuration is involved, changing the timezone to non-UTC yields an error. Removes the check plugin since the jar dependency is built with a newer JVM version and fails the build.

src/main/java/io/lenses/connect/smt/header/InsertWallclock.java

src/main/java/io/lenses/connect/smt/header/InsertWallclockDateTimePart.java

GoMati-MU · 2024-02-23T12:45:39Z

src/main/java/io/lenses/connect/smt/header/InsertWallclockDateTimePart.java

@@ -81,7 +92,9 @@ public R apply(R r) {
      return null;
    }

-    final String value = valueExtractorF.apply(instantF.get());
+    Instant now = instantF.get();


how about nowSupplier or nowFunction instead of instantF?

src/main/java/io/lenses/connect/smt/header/InsertWallclockDateTimePart.java

src/main/java/io/lenses/connect/smt/header/InsertRollingWallclock.java

src/main/java/io/lenses/connect/smt/header/TimestampConverter.java

GoMati-MU · 2024-02-23T13:05:43Z

src/main/java/io/lenses/connect/smt/header/TimestampConverter.java

-            result.set(Calendar.SECOND, 0);
-            result.set(Calendar.MILLISECOND, 0);
-            return result.getTime();
+            ZonedDateTime truncated =


I may be missing a point here but why not just:

ZonedDateTime.ofInstant(orig.toInstant(), config.targetTimeZoneId);

or ZoneOffset.UTC.normalized() when you need UTC? Cause here you pass two zones - one from config and second is UTC.

But it's still much better than what was there before the change ;)

The translator is responsible for only taking the date part of the incoming Date.

So:

orig.toInstant().atZone(config.targetTimeZoneId).toLocalDate(): extracts the date based on the configured timezone

LocalTime.MIDNIGHT: it's all about the 0 hour, 0 minute, 0 second, etc

UTC: at this point we don't want another timezone translation because on 01-01 2024 0:0:0 Chicago time would be still in 2023

I see the point now, thanks! But IMO it's still too much of a hassle and could be more readable. My proposition:

ZonedDateTime truncated = ZonedDateTime.ofInstant(orig.toInstant(), config.targetTimeZoneId).truncatedTo(ChronoUnit.DAYS); return Date.from(truncated.toInstant());

The option you shared yields incorrect results. in this case it will return the data as of 7am 01/01/1970 - see test testSchemalessTimestampToDateOnNonUTC

GoMati-MU · 2024-02-23T13:09:40Z

src/main/java/io/lenses/connect/smt/header/TimestampConverter.java

-            result.set(Calendar.MILLISECOND, origCalendar.get(Calendar.MILLISECOND));
-            return result.getTime();
+            ZonedDateTime zonedDateTime = orig.toInstant().atZone(config.targetTimeZoneId);
+            return Date.from(


same as above :)

ZonedDateTime zonedDateTime = ZonedDateTime.ofInstant(orig.toInstant(), config.targetTimeZoneId); return Date.from(zonedDateTime.toInstant());

src/main/java/io/lenses/connect/smt/header/TimestampConverter.java

GoMati-MU · 2024-02-23T13:17:05Z

src/test/java/io/lenses/connect/smt/header/InsertWallclockDateTimePartTest.java

+    Map<String, String> configs = new HashMap<>();
+    configs.put("header.name", "wallclock");
+    configs.put("date.time.part", "year");
+    configs.put("timezone", "Asia/Kolkata");


MINOR: it's nice to pass comment with time difference for better understanding e.g.

String kolkataTimezone = "Asia/Kolkata"; // Kolkata is GMT+5:30 which makes it 2021 there. configs.put("timezone", kolkataTimezone);

Wouldn't the code assertion infer:

Kolkata is GMT+5:30 which makes it 2021 there.

I corrected my code :) Comment is one way to make developer know that TimeZone difference will cause Year to flip. Another is to add message to assertion e.g.

assertEquals("2021", transformed.headers().lastWithName("wallclock").value(), "Year should flip as we calculate it for Kolkata Timezone");