fix issue with segment metadata cache and complex types when doing out of order upgrades from 0.22 #14948

clintropolis · 2023-09-06T22:05:24Z

Description

Fixes an issue with complex type handling that was intended to be fixed as part of #12016 to handle cases where the broker is updated before historical and realtime nodes.

Since complex types were sending back the response with just the complex type name, the valueType was still null resulting in errors of the form:

org.apache.druid.java.util.common.ISE: Encountered null type for column [theta_user]
	at org.apache.druid.sql.calcite.schema.SegmentMetadataCache.lambda$buildDruidTable$13(SegmentMetadataCache.java:825) ~[classes/:?]
	at java.util.Optional.orElseThrow(Optional.java:403) ~[?:?]
...

I will update the release notes of 0.23 to indicate that the brokers must be updated last for now...

This PR has:

been self-reviewed.
a release note entry in the PR description.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
been tested in a test Druid cluster.

…t of order upgrades from 0.22

churromorales

LGTM

sql/src/main/java/org/apache/druid/sql/calcite/schema/SegmentMetadataCache.java

@@ -961,6 +961,9 @@
        // likelyhood of upgrading from some version lower than 0.23 is low
        try {
          valueType = ColumnType.fromString(entry.getValue().getType());
+          if (valueType == null) {
+            valueType = ColumnType.ofComplex(entry.getValue().getType());


kfaraz

Thanks for the fix!

fix issue with segment metadata cache and complex types when doing ou…

14d0c0d

…t of order upgrades from 0.22

clintropolis added the Bug label Sep 6, 2023

churromorales approved these changes Sep 6, 2023

View reviewed changes

github-advanced-security bot found potential problems Sep 6, 2023

View reviewed changes

kfaraz approved these changes Sep 7, 2023

View reviewed changes

asdf2014 approved these changes Sep 12, 2023

View reviewed changes

asdf2014 merged commit 5cecf6c into apache:master Sep 12, 2023

clintropolis deleted the fix-out-of-order-upgrade-from-22-complex branch September 12, 2023 03:09

LakshSingla added this to the 28.0 milestone Oct 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix issue with segment metadata cache and complex types when doing out of order upgrades from 0.22 #14948

fix issue with segment metadata cache and complex types when doing out of order upgrades from 0.22 #14948

clintropolis commented Sep 6, 2023

churromorales left a comment

kfaraz left a comment

fix issue with segment metadata cache and complex types when doing out of order upgrades from 0.22 #14948

fix issue with segment metadata cache and complex types when doing out of order upgrades from 0.22 #14948

Conversation

clintropolis commented Sep 6, 2023

Description

churromorales left a comment

Choose a reason for hiding this comment

kfaraz left a comment

Choose a reason for hiding this comment