Annotation Context

Overview

Any visualization that assigns an identifier ("Class ID") to an instance or entity can benefit from using Annotations. By using an Annotation Context, you can associate labels and colors with a given class and then re-use that class across entities.

This is particularly useful for visualizing the output of classifications algorithms (as demonstrated by the Detect and Track Objects example), but can be used more generally for any kind of reoccurring categorization within a Rerun recording.

Keypoints & keypoint connections

Rerun allows you to define keypoints within a class. Each keypoint can define its own properties (colors, labels, etc.) that overwrite its parent class.

A typical example usage of keypoints is annotating the joints of a skeleton within a pose detection. In that case, the entire detected pose/skeleton is assigned a Class ID and each joint within gets a Keypoint ID.

To help you more with this (and similar) use-case(s), you can also define connections between keypoints as part of your annotation class description. The viewer will draw the connecting lines for all connected keypoints whenever that class is used. Just as with labels and colors this allows you to use the same connection information on any instance that class in your scene.

Keypoints are currently only applicable to 2D and 3D points.

Logging an annotation context

Annotation Context is typically logged as timeless data, but can change over time if needed.

The Annotation Context is defined as a list of Class Descriptions that define how classes are styled (as well as optional keypoint style and connection).

Annotation contexts are logged with:

• Python: 🐍rr.AnnotationContext
• Rust: 🦀rerun::AnnotationContext

// Annotation context with two classes, using two labeled classes, of which ones defines a color.
MsgSender::new("masks") // Applies to all entities below "masks".
    .with_timeless(true)
    .with_component(&[AnnotationContext {
        class_map: [
            ClassDescription {
                info: AnnotationInfo {
                    id: 0,
                    label: Some(Label("Background".into())),
                    color: None,
                },
                ..Default::default()
            },
            ClassDescription {
                info: AnnotationInfo {
                    id: 0,
                    label: Some(Label("Person".into())),
                    color: Some(Color(0xFF000000)),
                },
                ..Default::default()
            },
        ]
        .into_iter()
        .map(|class| (ClassId(class.info.id), class))
        .collect(),
    }])?
    .send(rec)?;

// Annotation context with simple keypoints & keypoint connections.
MsgSender::new("detections") // Applies to all entities below "detections".
    .with_timeless(true)
    .with_component(&[AnnotationContext {
        class_map: std::iter::once((
            ClassId(0),
            ClassDescription {
                info: AnnotationInfo {
                    id: 0,
                    label: Some(Label("Snake".into())),
                    color: None,
                },
                keypoint_map: (0..10)
                    .map(|i| AnnotationInfo {
                        id: i,
                        label: None,
                        color: Some(Color::from_rgb(0, (255 / 9 * i) as u8, 0)),
                    })
                    .map(|keypoint| (KeypointId(keypoint.id), keypoint))
                    .collect(),
                keypoint_connections: (0..9)
                    .map(|i| (KeypointId(i), KeypointId(i + 1)))
                    .collect(),
            },
        ))
        .collect(),
    }])?
    .send(rec)?;

Affected entities

Each entity that uses a Class ID component (and optionally Keypoint ID components) will look for the nearest ancestor that in the entity path hierarchy that has an Annotation Context defined.

Segmentation images

Segmentation images are single channel integer images/tensors where each pixel represents a class id. By default, Rerun will automatically assign colors to each class id, but by defining an Annotation Context, you can explicitly determine the color of each class.

• Python: rr.SegmentationImage
• Rust: Log a rerun::SegmentationImage