fix: table read query deduplicates identical rows

[gtarpenning]

New table query introduced in: #2534 does not do row deduplication.

This pr

• introduces a new test that is breaking in prod
• adds deduplication to the table query on row number

Previously, we used a subquery for de-duplification of table rows by project_id, digest. See the old query here:

SELECT tr.digest, tr.val_dump
FROM (
    SELECT project_id, row_digest
    FROM (
        SELECT *
        FROM (
                SELECT *,
                    row_number() OVER (PARTITION BY project_id, digest) AS rn  -- <<<<<<
                FROM tables
                WHERE project_id = {project_id:String} AND digest = {digest:String}
            )
        WHERE rn = 1 -- <<<<<<
        ORDER BY project_id, digest
    )
    ARRAY JOIN row_digests AS row_digest
    WHERE digest = {digest:String}
) AS t

But now, in prod, we do not do this deduplification, instead relying on a DISTINCT clause. However, we now use row_order which does not have the same properties as row_number, and we no longer return unique digests. Here is the current query:

SELECT DISTINCT tr.digest, tr.val_dump, t.row_order
    FROM table_rows tr
    INNER JOIN (
        SELECT row_digest, row_number() OVER () AS row_order
        FROM (
            SELECT {row_digests_selection} as row_digests
            FROM tables
            WHERE project_id = {{{project_id_name}: String}}
            AND digest = {{{digest_name}: String}}
        )
        ARRAY JOIN row_digests AS row_digest
    ) AS t ON tr.digest = t.row_digest
    WHERE tr.project_id = {{{project_id_name}: String}} 
    ORDER BY row_order ASC

In this pr we make the minimally invasive change of adding LIMIT 1 to the subquery, which restricts identical rows to the first occurence

Performance:

tldr: identical performance

Query runs in prod, on a small table that requires deduplication (but is not getting it properly):

1. Elapsed: 0.124s. Read: 50,071 rows (81.28 MB)
2. Elapsed: 0.126s. Read: 50,071 rows (81.28 MB)
3. Elapsed: 0.110s. Read: 50,071 rows (81.28 MB)

This pr, on the same table, returning the correct result:

1. Elapsed: 0.099s. Read: 50,071 rows (81.28 MB)
2. Elapsed: 0.102s. Read: 50,071 rows (81.28 MB)
3. Elapsed: 0.093s. Read: 50,071 rows (81.28 MB)

Testing

• Adds test when recreating an identical dataset. The act of re-creating an identical dataset inserts identical rows. This test ensures that when we read the dataset we get the original set of rows.
• We want to be careful we aren't too aggressive in our deduplication, but the test test_table_query_with_duplicate_row_digests should cover our bases there.
• Manual testing via SDK:

In prod:

recreating dataset
📦 Published to https://wandb.ai/griffin_wb/dedupe-prod/weave/objects/grammar-3/versions/p2BSs16ikm1jQGdSufWL2cBRXVrXUxeO5ZYHdPOzPhw
WeaveObject(ObjectRecord({'name': 'grammar-3', 'description': None, 'rows': <weave.trace.vals.WeaveTable object at 0x3229ecb20>, '_class_name': 'Dataset', '_bases': ['Object', 'BaseModel']}))
WeaveDict({'id': '0', 'sentence': 'He no likes ice cream.', 'correction': "He doesn't like ice cream."})
WeaveDict({'id': '1', 'sentence': 'She goed to the store.', 'correction': 'She went to the store.'})
WeaveDict({'id': '2', 'sentence': 'They plays video games all day.', 'correction': 'They play video games all day.'})
WeaveDict({'id': '0', 'sentence': 'He no likes ice cream.', 'correction': "He doesn't like ice cream."})
WeaveDict({'id': '1', 'sentence': 'She goed to the store.', 'correction': 'She went to the store.'})
WeaveDict({'id': '2', 'sentence': 'They plays video games all day.', 'correction': 'They play video games all day.'})

In branch:

recreating dataset
📦 Published to https://wandb.ai/griffin_wb/dedupe-prod/weave/objects/grammar-3/versions/p2BSs16ikm1jQGdSufWL2cBRXVrXUxeO5ZYHdPOzPhw
WeaveObject(ObjectRecord({'name': 'grammar-3', 'description': None, 'rows': <weave.trace.vals.WeaveTable object at 0x3200ec9d0>, '_class_name': 'Dataset', '_bases': ['Object', 'BaseModel']}))
WeaveDict({'id': '0', 'sentence': 'He no likes ice cream.', 'correction': "He doesn't like ice cream."})
WeaveDict({'id': '1', 'sentence': 'She goed to the store.', 'correction': 'She went to the store.'})
WeaveDict({'id': '2', 'sentence': 'They plays video games all day.', 'correction': 'They play video games all day.'})

[circle-job-mirror]

Preview this PR with FeatureBee: https://beta.wandb.ai/?betaVersion=795627bf407c0478f187ad092dd7d0837c708b81

[tssweeney]

Since you don't actually care about ordering, i think you can just do LIMIT 1 in the inner query in both cases and reduce the overall cost of this query