-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: fixed nondeterminism in UdfIndex #7719
fix: fixed nondeterminism in UdfIndex #7719
Conversation
@confluentinc It looks like @Sullivan-Patrick just signed our Contributor License Agreement. 👍 Always at your service, clabot |
ksqldb-common/src/test/java/io/confluent/ksql/function/UdfIndexTest.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @Sullivan-Patrick ! LGTM besides the failing unit test and updating the "breaking change" description as we discussed offline. Great stuff.
ksqldb-common/src/main/java/io/confluent/ksql/function/UdfIndex.java
Outdated
Show resolved
Hide resolved
ksqldb-common/src/main/java/io/confluent/ksql/function/UdfIndex.java
Outdated
Show resolved
Hide resolved
ksqldb-common/src/main/java/io/confluent/ksql/function/UdfIndex.java
Outdated
Show resolved
Hide resolved
ksqldb-common/src/test/java/io/confluent/ksql/function/UdfIndexTest.java
Outdated
Show resolved
Hide resolved
ksqldb-common/src/test/java/io/confluent/ksql/function/UdfIndexTest.java
Outdated
Show resolved
Hide resolved
ksqldb-common/src/test/java/io/confluent/ksql/function/UdfIndexTest.java
Outdated
Show resolved
Hide resolved
b5b3125
to
be17dd6
Compare
ksqldb-common/src/test/java/io/confluent/ksql/function/UdfIndexTest.java
Outdated
Show resolved
Hide resolved
1cf3256
to
5c24e93
Compare
5c24e93
to
c2408a0
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @Sullivan-Patrick . This is great! One minor comment inline.
Also a suggestion to improve the "breaking change" message in your PR description (which is also great, btw): let's clarify what it means for "Functions that relied on vague implicit casting" to "no longer work." To "no longer work" means that any new queries will be rejected, and any existing queries will not be started after an upgrade.
LGTM otherwise!
ksqldb-common/src/main/java/io/confluent/ksql/function/UdfIndex.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @Sullivan-Patrick ! 🎉
Description
When working on
LEAST
andGREATEST
functions, I noticed there was some nondeterminism inside theUdfIndex
class, specifically with theorder
field, which was intended to give comparably equal functions a final ordering based in the way they were added inside the Udf's class. This wasn't working in practice, and was leading to equally comparable methods that matched a query being selected arbitrarily. For example, when callingLEAST(INT, LONG)
, several UDFs could be selected:LEAST(LONG, LONG...)
,LEAST(DOUBLE, DOUBLE...)
,LEAST(DECIMAL, DECIMAL)
.To fix this,
UdfIndex.Node.compare()
was changed to no longer considerorder
. As this change implies we are no longer able to select a matching function if multiple exist,getFunction
now throws aKsqlException
if there is more than one equally comparable method that would match the query.A minor change that was made on top of this is adding to
UdfIndex.Node.compare()
the preference of picking methods with fewer generics as a final comparison.BREAKING CHANGE: Existing queries that relied on vague implicit casting will not be started after an upgrade, and new queries that rely on vague implicit casting will be rejected. For example,
foo(INT, INT)
will not be able to resolve against two underlying function signatures offoo(BIGINT, BIGINT)
andfoo(DOUBLE, DOUBLE)
. Calling a function whose only parameter is variadic with an explicit null will also result in the call being rejected as vague.Testing done
Added to UDFIndexTest negative tests which ensure that queries which would rely on vague implicit casting fail properly.
Some tests that relied on vague implicit casting may need to be removed, this is worth discussion.
Reviewer checklist