Skip to navigation

Multi-Camera Tracking

Track several cameras mounted on one rig with a single shared map
Open in ClaudeOpen in ChatGPT

Since ZED SDK 5.6, several ZED and ZED One cameras mounted on the same rigid body can be tracked together, as one rig, with a single shared map. This multi-camera SLAM runs through the Sensors API (sl::Sensors), with PositionalTrackingSensorsParameters::multi_sensor_mode set to MULTI_SENSOR_TRACKING_MODE::SINGLE_BODY.

The Sensors API is in beta: its interfaces and behavior may change in future ZED SDK releases. Multi-camera tracking is available in the C++ and Python APIs.

Tracking Modes

ModeBehavior
MULTI_SENSOR_TRACKING_MODE::AUTODefault. The ZED SDK chooses; it currently resolves to PER_SENSOR.
MULTI_SENSOR_TRACKING_MODE::PER_SENSOREach ZED stereo camera tracks on its own, with its own map.
MULTI_SENSOR_TRACKING_MODE::SINGLE_BODYThe ZED stereo and ZED One cameras are tracked as one rig with a single shared map.

Sensors::getMultiSensorTrackingMode() returns the mode the tracking actually runs in.

Requirements

  • POSITIONAL_TRACKING_MODE::GEN_3.
  • At least one ZED stereo camera, and up to 8 cameras in total. LiDARs cannot be part of the rig.
  • All cameras at the same resolution and frame rate, sharing one CUDA context (see InitSensorsParameters::sdk_cuda_ctx).
  • Hardware-synchronized cameras, whose simultaneous images are captured within 1.5 ms of each other. This is checked on the first images: if it fails, an error is logged and the pose stays POSITIONAL_TRACKING_STATE::UNAVAILABLE. See Frame synchronization.
  • The pose of each camera on the rig, set with Sensors::setSensorPose() before enabling the tracking. It is expressed in the BASELINK frame of the rig, in the coordinate_units and coordinate_system of InitSensorsParameters.

enablePositionalTracking() returns SENSORS_ERROR_CODE::INVALID_FUNCTION_PARAMETERS when the mode is not GEN_3, when the rig contains a LiDAR, no ZED stereo camera or more than 8 cameras, when the cameras do not share one resolution, frame rate and CUDA context, or when tracking_camera_id is not a ZED stereo camera of the rig. The hardware synchronization is checked on the first images instead, as described above.

How It Works

  • One ZED stereo camera, set by PositionalTrackingSensorsParameters::tracking_camera_id (by default the first ZED stereo camera added), drives the tracking of the rig: its IMU is the one fused, and its pose places the rig in the world.
  • Each ZED One adds its images to the shared map and moves with the rig through its sensor pose.
  • The tracking of each sensor starts at its sensor pose, so the WORLD frame matches the BASELINK frame at start, unless you set initial_world_transforms. In SINGLE_BODY mode, only the initial_world_transforms entry of the tracking camera is used.
  • getPosition() in SENSORS_REFERENCE_FRAME::WORLD returns the pose of a sensor in the world frame. SENSORS_REFERENCE_FRAME::SENSOR returns its motion since the previous frame in its own axes, and SENSORS_REFERENCE_FRAME::BASELINK the same motion in the BASELINK axes.
  • saveAreaMap() saves the shared map, with the position of each camera on the rig. Area files in this format cannot be loaded by ZED SDK 5.5 or earlier.

Using the API

sl::Sensors sensors;
sl::InitSensorsParameters init_parameters;
init_parameters.coordinate_units = sl::UNIT::METER;
sensors.init(init_parameters);
// Add the cameras of the rig: hardware-synchronized, same resolution and frame rate
sl::SensorDeviceIdentifier zed_id, zed_one_id;
sl::InitParameters zed_parameters;
zed_parameters.input.setFromSerialNumber(zed_serial_number);
sensors.add(zed_parameters, zed_id);
sl::InitParametersOne zed_one_parameters;
zed_one_parameters.input.setFromSerialNumber(zed_one_serial_number);
sensors.add(zed_one_parameters, zed_one_id);
// Pose of each camera on the rig, in the BASELINK frame, set before enabling the tracking
sensors.setSensorPose(zed_pose_on_rig, zed_id);
sensors.setSensorPose(zed_one_pose_on_rig, zed_one_id);
sl::PositionalTrackingSensorsParameters tracking_parameters;
tracking_parameters.mode = sl::POSITIONAL_TRACKING_MODE::GEN_3;
tracking_parameters.multi_sensor_mode = sl::MULTI_SENSOR_TRACKING_MODE::SINGLE_BODY;
tracking_parameters.tracking_camera_id = zed_id;
sensors.enablePositionalTracking(tracking_parameters);
sl::Pose pose;
while (sensors.process() == sl::SENSORS_ERROR_CODE::SUCCESS) {
// Pose of the tracking camera in the world frame
sl::POSITIONAL_TRACKING_STATE state = sensors.getPosition(pose, sl::SENSORS_REFERENCE_FRAME::WORLD, zed_id);
}

Using a Motion Prior

Sensors::ingestMotionPrior() feeds your own motion source to the Sensors API tracking, like Camera::ingestMotionPrior(). The difference is RelativePoseData::sensor_transform: it is the pose of your source in the BASELINK frame, not in the frame of a camera.

  • In SINGLE_BODY mode, the motion is given to the rig: call it once per sample, with the default identifier or any camera of the rig.
  • In PER_SENSOR mode, the motion is given to the camera named by sensors_id only: call it for each camera that should use the source.

Sample

ZED SDK 5.6 adds a C++ sample for this mode. It runs from live cameras, SVO files or streams listed in a YAML file and shows the rig, its path and the shared map in 3D. See the Positional Tracking samples.