Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MAP keys serialize intederministically #6621

Closed
agavra opened this issue Nov 16, 2020 · 0 comments · Fixed by #6654
Closed

MAP keys serialize intederministically #6621

agavra opened this issue Nov 16, 2020 · 0 comments · Fixed by #6654
Assignees
Labels

Comments

@agavra
Copy link
Contributor

agavra commented Nov 16, 2020

Different implementations of Java's Map serialize differently. This means that we could have the same map serialize to two different bytes representations, and therefore get sent to different partitions. This was discovered with the following pull query test:

Note: this test will fail in master because we have not yet implemented non-literal keys. It was run on a test branch where I am implementing that feature

    {
      "name": "non-windowed - map keys",
      "properties": {
        "ksql.key.format.enabled": true
      },
      "statements": [
        "CREATE STREAM INPUT (ID MAP<STRING, STRING> KEY, IGNORED INT) WITH (kafka_topic='test_topic', format='JSON');",
        "CREATE TABLE AGGREGATE AS SELECT ID, COUNT(1) AS COUNT FROM INPUT GROUP BY ID;",
        "SELECT ID, COUNT, ROWTIME FROM AGGREGATE WHERE ID=MAP('F2':='0', 'F1':='1');"
      ],
      "inputs": [
        {"topic": "test_topic", "timestamp": 12345, "key": {"F1": "1", "F2": "1"}, "value": {}},
        {"topic": "test_topic", "timestamp": 12365, "key": {"F1": "1", "F2": "0"}, "value": {}}
      ],
      "responses": [
        {"admin": {"@type": "currentStatus"}},
        {"admin": {"@type": "currentStatus"}},
        {"query": [
          {"header":{"schema":"`ID` MAP<STRING, STRING> KEY, `COUNT` BIGINT, `ROWTIME` BIGINT"}},
          {"row":{"columns":[{"F1": "1", "F2": "0"}, 1, 12365]}}
        ]}
      ]
    },

The lookup could not succeed because the serialized key did not match the serialized where clause value.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants