Skip to navigation

Multi-camera setup

Open in ClaudeOpen in ChatGPT

In this tutorial you will learn how to create a multi-camera configuration with ZED devices.

For a multi-camera configuration it is important to correctly identify each camera in the system by using its serial number and precisely set the position and the orientation of each of them with respect to a reference point.

A working example is available in the zed-ros2-examples GitHub repository.

The launch file explained

The example launch file zed_multi_camera.launch.py allows you to dynamically configure a robotics system equipped with multiple different models of ZED cameras.

The launch file starts one Robot State Publisher node that defines the position and orientation of each camera in the multi-camera system, one Robot State Publisher node for each camera that broadcasts all the static frames of the camera, and one ZED node for each camera of the system.

The number of ZED nodes to start is inferred from the size of the parameters passed to the launch file from the command line.

All the ZED nodes are loaded in the same ROS 2 container as components using ROS 2 Composition.

Do not use this launch file to record SVO files from several cameras at the same time. On NVIDIA® Jetson™, recording several cameras loaded in the same container can make the hardware encoder fail after a few minutes and stop all the cameras. To record SVO files, run each camera in its own container, as explained in Record and Replay Camera Data.

Launch parameters

  • cam_names: (REQUIRED) An array containing the name of the cameras, e.g. '[zed_front,zed_rear]'. Each camera must have a different name, so that the cameras can be told apart.
  • cam_models: (REQUIRED) An array containing the model of the cameras, e.g. '[zed2i,zed2]'
  • cam_serials: (REQUIRED, unless cam_ids or sim_mode is used) An array containing the serial numbers of the cameras, e.g. [3001234,2001234]
  • cam_ids: An array containing the IDs of the cameras, e.g. [0,1], used in place of cam_serials.
  • sim_mode: set it to true when the cameras are streamed by a simulator, such as NVIDIA® Isaac™ Sim. sim_ports then replaces cam_serials and cam_ids.
  • sim_ports: in simulation mode, an array containing the streaming port of each simulated camera, e.g. '[30000,30002]'.
  • sim_address: the address of the simulation server, 127.0.0.1 by default.
  • use_sim_time: use the simulation time from the /clock topic.
  • disable_tf: Only the first camera is configured to broadcast the TF map -> odom -> camera_link to avoid TF conflicts. The TF broadcasting can be disabled by setting this parameter to true. This is required if an external Kalman filter that fuses different odometry sources is used.

All the arrays must have the same size.

Run the example

Launch a dual camera node example by using the following command

ros2 launch zed_multi_camera zed_multi_camera.launch.py cam_names:='[zed_front,zed_rear]' cam_models:='[zedx,zedxm]' cam_serials:='[<serial_camera_front>,<serial_camera_rear>]'

Info: to retrieve the serial number of each connected camera you can use the command ZED_Explorer --all.

Simulated cameras

Since v5.6.0 of the ROS 2 Wrapper, the example also starts several cameras streamed by a simulator, such as NVIDIA® Isaac™ Sim with the ZED extension: set sim_mode:=true and give the streaming port of each simulated camera in sim_ports:

ros2 launch zed_multi_camera zed_multi_camera.launch.py cam_names:='[zed_front,zed_rear]' cam_models:='[zedx,zedx]' sim_mode:=true sim_ports:='[30000,30002]'

With ZED SDK >= 5.6 on both the simulator and the ROS 2 side, any number of simulated cameras is supported: each one has its own serial number and calibration, so no serial number or calibration file has to be configured. See Using the ZED with ROS 2 in Isaac™ Sim to set up the simulated cameras.

The Multi Camera URDF

It is important to create a URDF that defines the position of all the cameras with respect to a reference link.

We provide an example in ‘urdf/zed_multi.urdf.xacro’.

The URDF is created by using the ‘xacro’ tool.

First of all, reference links are created:

<link name="$(arg multi_link)" />
<link name="$(arg camera_name_0)_camera_link" />
<link name="$(arg camera_name_1)_camera_link" />

The name of the reference link is contained in the xacro argument multi_link, the default value is zed_multi_link. A “virtual” link camera_link for each camera must be defined to correctly connect the URDF of each camera to the reference link.

Next a joint to connect the reference link to the main link of each camera is created:

<joint name="$(arg camera_name_0)_camera_joint" type="fixed">
<parent link="$(arg camera_name_0)_camera_link"/>
<child link="$(arg multi_link)"/>
<origin xyz="0.06 0.0 0.0" rpy="0 0 0" />
</joint>
<joint name="$(arg camera_name_1)_camera_joint" type="fixed">
<parent link="$(arg multi_link)"/>
<child link="$(arg camera_name_1)_camera_link"/>
<origin xyz="-0.06 0.0 0.0" rpy="0 0 ${M_PI}" />
</joint>

The name of the joint is created from the variable camera_name_x that identifies the name of each camera. A position and orientation must be provided for each camera with respect to the main link:

<origin xyz="x y z" rpy="r p y" />

the position is in meters, the orientation in radians.

The first joint has parent link and child link inverted with respect to all the other joints. This is required because the first camera is the reference for visual odometry processing and in ROS a joint cannot have two parents.

The following image displays the TF tree generated for the dual camera configuration of this tutorial:

image

You can retrieve the name of the container and check the components that are running in the same process by using the command

ros2 component list

for example:

ros2 component list
/zed_multi/zed_multi_container
1 /zed_multi/zed_front
2 /zed_multi/zed_rear

All the nodes run in the same namespace zed_multi.

Synchronized publishing of GMSL cameras

GMSL cameras connected to the same capture card capture their frames at the same instant. Since v5.6.0 of the ROS 2 Wrapper, they can also publish the same captures. It works with the stereo node (ZED X family) and the mono node (ZED X One family), also mixed in the same system. This matters when the publishing rate is lower than the grab rate (general.pub_frame_rate < general.grab_frame_rate), for example to reduce the load of a multi-camera robot.

By default, each ZED node selects the frames to publish with its own publishing timer. The timers of the different nodes are not related to each other, so two cameras that capture at the same instant usually publish frames captured one or more grab periods apart: with general.grab_frame_rate: 30 and general.pub_frame_rate: 10, the images published by two cameras can be 33 ms, 67 ms or more apart, and the gap can change while the nodes run.

With general.sync_pub_to_timestamp: true, each node selects the frames to publish on their image timestamp instead: time is divided into publishing periods (100 ms at 10 Hz) and the node publishes the first frame captured in each period. The synchronized cameras share the same capture timestamps, so all the nodes select the same frames, without any communication between them, and their images, depth maps and point clouds carry matching timestamps.

How to enable it

Set the parameter in the common_stereo.yaml file for the stereo cameras and in the common_mono.yaml file for the ZED X One cameras:

general:
grab_frame_rate: 30
pub_frame_rate: 10.0
sync_pub_to_timestamp: true

Requirements:

  • The cameras must capture at the same instant: GMSL cameras connected to the same capture card.
  • All the nodes must use the same general.grab_frame_rate and the same general.pub_frame_rate.

What changes when it is enabled

The mono node only changes how it selects the frames to publish. In the stereo node, which also computes the depth:

  • The camera is grabbed at general.grab_frame_rate: general.grab_compute_capping_fps is ignored, with a warning at start-up if it is set.
  • The point cloud is published at general.pub_frame_rate, together with the images: depth.point_cloud_freq is not used.
  • When the depth is only used for publishing, the node computes it only for the published frames, so the GPU load follows general.pub_frame_rate instead of general.grab_frame_rate. When a module needs the depth of every frame (Object Detection, Body Tracking, Spatial Mapping, or the positional tracking in a mode other than GEN_3), the depth is computed for every frame as usual.
  • If the depth of the frame to publish is missing (for example, a frame was dropped), the node publishes the next frame and logs it: for that period only, the camera publishes one grab period later than the others.

In a test on an NVIDIA® Jetson AGX Orin™ (grab_frame_rate: 30, pub_frame_rate: 10) with three pairs of cameras (two ZED X Mini, two ZED X One GS, and a ZED X Mini with a ZED X One GS), the cameras published frames captured one grab period (33 ms) apart without synchronized publishing, or by chance the same captures, depending on when the nodes started. With sync_pub_to_timestamp: true, more than 99% of the published pairs came from the same capture in every combination.

Leave the parameter set to false (the default) for a single camera, or for cameras that do not capture at the same instant: it does not synchronize the capture of the cameras, it only makes the publishing follow it.