Make llm benchmarks query faster

[clee2000]

Make faster by adding a projection to the table

PROJECTION benchmark_name_projection
    (
        SELECT *
        ORDER BY 
            repo,
            tupleElement(benchmark, 'name'),
            tupleElement(model, 'name'),
            tupleElement(metric, 'name'),
            timestamp,
            head_branch,
            head_sha,
            workflow_id,
            job_id,
            servicelab_experiment_id,
            servicelab_trial_id
    ),

For some reason, ClickHouse won't use the projection unless t uses has instead of in and all the other tuple elements are in quotation marks

We should consider changing the order by key to be this instead since I can't imagine anyone not filtering by benchmark name

Perf results:

+------+----------+-----------+-------------+---------------+----------+-----------+------------+--------------+
| Test | Avg Time | Base Time | Time Change | % Time Change | Avg Mem  |  Base Mem | Mem Change | % Mem Change |
+------+----------+-----------+-------------+---------------+----------+-----------+------------+--------------+
|  0   |   131    |   33300   |    -33169   |      -100     | 57215999 | 335780375 | -278564376 |     -83      |
+------+----------+-----------+-------------+---------------+----------+-----------+------------+--------------+

Also added a test

Checked results were the same after sorting

Also a change to the script to compare queries that makes it easier to compare results that are different by piping them to files that can be diffed later

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
torchci	✅ Ready (Inspect)	Visit Preview	Feb 5, 2025 0:30am

[huydhn]

One of of confusing aspect of ClickHouse is that sometimes model.name works, sometimes it needs to be model.'name', and the other times it is tupleElement(model, 'name'). The last one is probably the clearest because the field is a tuple, but it's also the lengthiest

[huydhn]

LGTM! Thank you for the fix!

[huydhn]

I'm not sure why the preview doesn't show up because Vercel cancels it

[huydhn]

I think we should also add some more key to the projection if possible. They are the keys that I should have added when creating the table.

PROJECTION benchmark_name_projection
    (
        SELECT *
        ORDER BY 
            repo,
            tupleElement(benchmark, 'name'),
            tupleElement(model, 'name'),
            tupleElement(metric, 'name'),
            timestamp,
            head_branch,
            head_sha,
            workflow_id,
            job_id,
            servicelab_experiment_id,
            servicelab_trial_id
    ),

[huydhn]

Also, could you try update the two queries?

• https://github.com/pytorch/test-infra/tree/main/torchci/clickhouse_queries/oss_ci_benchmark_branches
• and https://github.com/pytorch/test-infra/tree/main/torchci/clickhouse_queries/oss_ci_benchmark_names

I'm currently using a materialize view for them, which I think is slower than this projection approach (when I try to load the page)

It's better done in a separate PR then, as I think #6167 needs to be reverted to try it out