Example of using Rerun to log and visualize the output of Meta AI's Segment Anything model.

Used Rerun types

Image, Tensor, SegmentationImage, Boxes2D

Background

This example showcases the visualization capabilities of Meta AI's Segment Anything model. The visualization provided in this example demonstrates the precise and accurate segmentation capabilities of the model, effectively distinguishing each object from the background and creating a transparent mask around them.

Logging and visualizing with Rerun

The visualizations in this example were created with the following Rerun code:

Timelines

All data logged using Rerun in the following sections is connected to a specific frame. Rerun assigns a frame to each piece of logged data, and these timestamps are associated with a timeline.

for n, image_uri in enumerate(args.images):
   rr.set_time_sequence("image", n)
   image = load_image(image_uri)
   run_segmentation(mask_generator, image)

Image

The input image is logged as Image to the image entity.

rr.log("image", rr.Image(image))

Segmentation

All masks are stacked together and logged using the Tensor archetype.

rr.log("mask_tensor", rr.Tensor(mask_tensor))

Then, all the masks are layered together and the result is logged as a SegmentationImage to the image/masks entity.

rr.log("image/masks", rr.SegmentationImage(segmentation_img.astype(np.uint8)))

For object localization, bounding boxes of segmentations are logged as Boxes2D.

rr.log(
    "image/boxes",
    rr.Boxes2D(array=mask_bbox, array_format=rr.Box2DFormat.XYWH, class_ids=[id for id, _ in masks_with_ids]),
)

Run the code

To run this example, make sure you have the Rerun repository checked out and the latest SDK installed:

# Setup
pip install --upgrade rerun-sdk  # install the latest Rerun SDK
git clone git@github.com:rerun-io/rerun.git  # Clone the repository
cd rerun
git checkout latest  # Check out the commit matching the latest SDK release

Install the necessary libraries specified in the requirements file:

pip install -r examples/python/segment_anything_model/requirements.txt

To experiment with the provided example, simply execute the main Python script:

python examples/python/segment_anything_model/main.py # run the example

If you wish to customize it or explore additional features, use the CLI with the --help option for guidance:

python examples/python/segment_anything_model/main.py --help