Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ChunkStore: implement new component-less indices and APIs #6879

Merged
merged 7 commits into from
Jul 15, 2024

Conversation

teh-cmc
Copy link
Member

@teh-cmc teh-cmc commented Jul 12, 2024

Introduces new component-less indices and APIs that allow for efficiently searching for relevant chunks directly at the entity level, as opposed to the entity+component level.

cc @jprochazk

/// Returns the most-relevant _temporal_ chunk(s) for the given [`LatestAtQuery`].
///
/// The returned vector is guaranteed free of duplicates, by definition.
///
/// The [`ChunkStore`] always work at the [`Chunk`] level (as opposed to the row level): it is
/// oblivious to the data therein.
/// For that reason, and because [`Chunk`]s are allowed to temporally overlap, it is possible
/// that a query has more than one relevant chunk.
///
/// The caller should filter the returned chunks further (see [`Chunk::latest_at`]) in order to
/// determine what exact row contains the final result.
///
/// **This ignores static data.**
pub fn latest_at_relevant_chunks_for_all_components(
    &self,
    query: &LatestAtQuery,
    entity_path: &EntityPath,
) -> Vec<Arc<Chunk>> {
    // ...
}

/// Returns the most-relevant _temporal_ chunk(s) for the given [`RangeQuery`].
///
/// The returned vector is guaranteed free of duplicates, by definition.
///
/// The criteria for returning a chunk is only that it may contain data that overlaps with
/// the queried range.
///
/// The caller should filter the returned chunks further (see [`Chunk::range`]) in order to
/// determine how exactly each row of data fit with the rest.
///
/// **This ignores static data.**
pub fn range_relevant_chunks_for_all_components(
    &self,
    query: &RangeQuery,
    entity_path: &EntityPath,
) -> Vec<Arc<Chunk>> {
    // ...
}

Checklist

  • I have read and agree to Contributor Guide and the Code of Conduct
  • I've included a screenshot or gif (if applicable)
  • I have tested the web demo (if applicable):
  • The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG
  • If applicable, add a new check to the release checklist!
  • If have noted any breaking changes to the log API in CHANGELOG.md and the migration guide

To run all checks from main, comment on the PR with @rerun-bot full-check.

@teh-cmc teh-cmc added ⛃ re_datastore affects the datastore itself 🔍 re_query affects re_query itself 🚀 performance Optimization, memory use, etc include in changelog labels Jul 12, 2024
Copy link
Member

@jprochazk jprochazk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested by temporarily merging into my PR, works well

@teh-cmc teh-cmc merged commit 528eda6 into main Jul 15, 2024
34 checks passed
@teh-cmc teh-cmc deleted the cmc/store_allcomponent_index branch July 15, 2024 10:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
include in changelog 🚀 performance Optimization, memory use, etc ⛃ re_datastore affects the datastore itself 🔍 re_query affects re_query itself
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants