Simplify annotation context

[jleibs]

AnnotationContext is currently very general/flexible, but subsequently is one of our most confusing archetypes we have. It's also implemented as a single mono-component, which means it has a very challenging type to work with natively in arrow, requiring complex serializers and deserializers.

Current functionality

Annotation context currently does 3 things:

• Allows you to apply colors/labels to segmentation images and other archetypes based on class-id
• Allows you to apply alternative colors/labels to "keypoints" within a class based on (class_id, keypoint_id)
• Allows you to connect keypoints together based on a skeleton connectivity.

A particular complexity of the current design is that skeleton information is managed per-class-id.

Proposed simplification

• Drop KeypointId as a distinct concept from ClassId
  • Rationale is that very few users leverage the nested hierarchy for colors/labels.
  • Those that do can still work around by creating a derived id by generating a combined id such as: (class_id << 8 | keypoint_id)
• Optional: we should consider renaming ClassId -> AnnotationId to make it more generic.
• Remove skeleton connectivity from ClassDescription
• Simplify AnnotationContext to be an archetype with 3 components: Ids, Colors, Labels.
• Introduce a new Skeleton / Edges Archetype to support use-cases where users want to visualize connectivity

Trade-offs

The main downside is that some of current "skeleton" use-cases will be more difficult:

• Users will have to manually build a skeleton based on INDEX offsets of their points rather than class-ids.
• This means logging the keypoints in a deterministic order so that these indicies are fixed.
• It also means that if some keypoints (and edges) might be missing, the skeleton will have to be dynamically adjusted on every frame.

One thing we could do to improve this would be optionally allowing edges to be specified by class-id rather than by index offset. This would just mean the viewer would do an on-the-fly remapping. This is a good candidate for a follow-on enhancement.

The big upside of this, however, unlocking a bunch of new graph edge visualizations that are currently not easy because AnnotationContext feels unnatural and adds too much complexity.

Additionally, treating edges as instances means unlocking new functionality like colored / labeled edges.

Implementation Details

AnnotationContext

The new archetype will look vaguely like:

archetype AnnotationContext {
  ids: Vec<AnnotationId>,
  colors: Vec<Color>,
  labels: Vec<Labels>
}

This means the AnnotationContext is a very simple Batch-style object. This can be easily represented as a table and should be easy for users to inspect and think about.

Nice-to-have: ideally the _ext.py would expose an API that allowed you to pass these as mappings and take care of everything else.

rr.log("/", AnnotationContext(labels: {1: "foo", 2: "bar"}))

Skeleton/Edges

Rather than be an implicit part of AnnotationContext, this will be a new standalone visualizers. It's not totally clear what to name this to optimize for discoverabillity. Some users will think of this as classic skeleton scaffolding, while others might be thinking more as a graph connectivity representation.

The new edge component will look very similar to TriangleIndices:

component EdgeIndices {
  indices: UVec2D
}

The new archetype will look vaguely like:

archetype Edges3D {
  # Per-instance components
  edges: Vec<EdgeIndices>
  colors: Vec<Color>,
  ...
  # Referenced components
  nodes: Vec<Position3D>,

Note that in this archetype each EDGE is an instance. This means you can apply colors/labels to the individual edges.

Implementation complexity:

This highlights a fundamental issue with our component model.

If both Edges and Points want to use the Color component, but have different interpretations of indexing, there is no way to reconcile them. One of them is going to get bad color-data no matter what you do.

[jleibs]

• Related: Improving on Annotation Context (meta) #2405