Dataframe queries 2: latest-at

[teh-cmc]

Implements the latest-api dataframe API.

Examples:

cargo r --all-features -p re_dataframe --example latest_at -- /tmp/helix.rrd
cargo r --all-features -p re_dataframe --example latest_at -- /tmp/helix.rrd /helix/structure/scaffolding/**

use itertools::Itertools as _;

use re_chunk::{TimeInt, Timeline};
use re_chunk_store::{ChunkStore, ChunkStoreConfig, LatestAtQueryExpression, VersionPolicy};
use re_dataframe::QueryEngine;
use re_log_types::StoreKind;

fn main() -> anyhow::Result<()> {
    let args = std::env::args().collect_vec();

    let get_arg = |i| {
        let Some(value) = args.get(i) else {
            eprintln!(
                "Usage: {} <path_to_rrd> <entity_path_expr>",
                args.first().map_or("$BIN", |s| s.as_str())
            );
            std::process::exit(1);
        };
        value
    };

    let path_to_rrd = get_arg(1);
    let entity_path_expr = args.get(2).map_or("/**", |s| s.as_str());

    let stores = ChunkStore::from_rrd_filepath(
        &ChunkStoreConfig::DEFAULT,
        path_to_rrd,
        VersionPolicy::Warn,
    )?;

    for (store_id, store) in &stores {
        if store_id.kind != StoreKind::Recording {
            continue;
        }

        let cache = re_dataframe::external::re_query::Caches::new(store);
        let engine = QueryEngine {
            store,
            cache: &cache,
        };

        let query = LatestAtQueryExpression {
            entity_path_expr: entity_path_expr.into(),
            timeline: Timeline::log_time(),
            at: TimeInt::MAX,
        };

        let query_handle = engine.latest_at(&query, None /* columns */);
        let batch = query_handle.get();

        eprintln!("{query}:\n{batch}");
    }

    Ok(())
}

• Part of New ChunkStore APIs to facilitate data access #7284

---

Dataframe APIs PR series:

• Dataframe queries 0: bootstrap & data model #7338
• Dataframe queries 1: schema resolution #7339
• Dataframe queries 2: latest-at #7340
• Dataframe queries 3: dense range #7341
• Dataframe queries 4: paginated dense range #7345

Checklist

• I have read and agree to Contributor Guide and the Code of Conduct
• I've included a screenshot or gif (if applicable)
• I have tested the web demo (if applicable):
  • Using examples from latest main build: rerun.io/viewer
  • Using full set of examples from nightly build: rerun.io/viewer
• The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG
• If applicable, add a new check to the release checklist!
• If have noted any breaking changes to the log API in CHANGELOG.md and the migration guide

• PR Build Summary
• Recent benchmark results
• Wasm size tracking

To run all checks from main, comment on the PR with @rerun-bot full-check.

[abey79]

It seems that schema_for_query() may return a false positive that could leak into LatestAtQueryHandle::schema()—unless I missed something, which is likely. If so, what happens in such a case?

[teh-cmc]

Yes, all schema-related methods are best-effort in that they only rely on chunk metadata rather than full-blown queries (this will become extremely important once we stop assuming that the entire index is available in local memory).

The schema of the result always matches perfectly with the announced schema: the false positives are simply filled with null values.