Using ZED with ROS 2 and Isaac™ Sim

This tutorial guides you through configuring the Isaac™ Sim ROS 2 Bridge to send simulated information to a ZED ROS 2 Wrapper node.

Install the dependencies

This installation guide is valid only for Ubuntu. For other operating systems, please refer to the official NVIDIA® documentation.

Some ROS 2 message types used by the Isaac™ Sim ROS 2 Bridge (such as Detection2DArray and Detection3DArray) depend on the Vision Messages ROS 2 package. We recommend installing it so that all message types are available:

$ sudo apt install ros-humble-vision-msgs

IMPORTANT: do not permanently append the bridge’s internal ROS 2 libraries to LD_LIBRARY_PATH in your ~/.bashrc. Doing so creates conflicts between libraries that prevent rviz2 and other ROS 2 tools from starting. Source your ROS 2 workspace once per terminal instead, as shown below.

For additional details, we recommend reading the official NVIDIA® documentation.

Set up the environment variables for ROS 2

Ensure that the ROS 2 environment is sourced in the terminal before starting Isaac™ Sim:

$ source /opt/ros/humble/setup.bash

You must run this command in the same terminal before launching Isaac™ Sim with ./isaac-sim.sh (located in the _build/linux-x86_64/release folder of your Isaac™ Sim build). Alternatively, add the line to your ~/.bashrc so the ROS 2 environment is configured automatically each time you open a terminal.

Enable the ROS 2 Bridge Extension

When Isaac™ Sim is ready, you must enable the isaacsim.ros2.bridge extension.

Open the Extensions view from the menu, Window -> Extensions.
Search for isaacsim.ros2.bridge using the search form in the top left.
Toggle the switch to enable the extension.
[Optional] Enable the AUTOLOAD option to skip this step each time you start Isaac™ Sim.

Enable the ZED Camera Extension

The ZED Camera extension is required to send the simulated ZED images to the ZED ROS 2 Wrapper.

Open the Extensions view from the menu, Window -> Extensions.
Search for ZED using the search form in the top left.
- Select the THIRD PARTY tab if you installed the extension from the GitHub repository (see the installation instructions for adding the extension if it is not available).
Toggle the switch to enable the extension.
[Optional] Enable the AUTOLOAD option to skip this step each time you start Isaac™ Sim.

Add a scenario to the Stage

The ZED camera is a visual sensor, so it’s important to add an environment that is rich in visual features for a good simulation experience.

Isaac™ Sim provides ready-made environments to use. From Create -> Isaac -> Environments, you can select, for example, the environment named Small Warehouse with Multiple Shelves.

Wait for the simulator to load the environment. This task can take a while, depending on the size of the environment that you selected.

Once all the objects in the scene are finally loaded, the simulator will display the selected environment from the default point of view of the viewport camera.

Add a ZED camera to the scene

The USD files for the ZED camera models are bundled with the ZED Camera extension, in the exts/sl.sensor.camera/data/usd folder. If you don’t already have the extension locally, clone the GitHub repository:

$ cd <your_workspace>
$ git clone git@github.com:stereolabs/zed-isaac-sim.git

Select the model ZED_X.usdc and drag it into the Viewport to place it in the position that you prefer.

A new prim named ZED_X will appear in the Stage view as a child of World.

The ZED_X prim has an important XForm named base_link; this is the reference frame of the mounting point on the bottom of the camera.

Enable the ZED camera

The data streaming is activated by using the OmniGraph node ZED Camera Helper.

Create a new Action Graph, Create -> Visual Scripting -> Action Graph.
Add a new node of type On Playback Tick to the graph. You can easily search for it by writing Tick in the search form.
Type ZED in the search form and drag the ZED Camera Helper node into the graph.
Connect the ExecIn input of ZED Camera Helper to the Tick output of the On Playback Tick node.

The ZED Camera extension also includes a lower-level ZED Stream node. For most use cases, make sure to use the ZED Camera Helper node (or the ZED Camera One Helper node for monocular ZED X One cameras).

Click on the ZED Camera Helper in the graph and select the Property tab on the right.
Click + Add Target near ZED Camera prim.
Select the root XForm named ZED_X and hit Select.

Publish simulation time to ROS 2

When working with simulated data, the ROS 2 nodes cannot rely on the system clock to timestamp the data accurately. Instead, they have to use the simulation time, which can be faster or slower than real time.

To handle this behavior, Isaac™ Sim ROS 2 Bridge provides a method to publish messages to the /clock topic with simulation time data.

All the ROS 2 nodes have a parameter named use_sim_time. When this parameter is set to true, each ROS 2 node subscribes to the /clock topic, and uses the simulation time as the reference for time instead of the system time.

Select the Action Graph tab in the bottom panel.
Add a new node of type Isaac Read Simulation Time.
- [Optional] Click on the Isaac Read Simulation Time.
- [Optional] Select the Properties tab in the right panel.
- [Optional] Check the Reset on stop option to reset the simulation time to zero each time the simulation is stopped and restarted.
Add a new node of type ROS2 Publish Clock.
Connect the Tick output of the On Playback Tick node to the ExecIn input of ROS2 Publish Clock.
Connect the Simulation Time output of the Isaac Read Simulation Time node to the Time Stamp input of ROS2 Publish Clock.

Start the simulation by hitting the Play button on the left panel

Now you can verify that the configuration is valid:

open a Terminal console (Ctrl + Alt + t).
enter the command ros2 topic echo /clock.
verify that the /clock topic is correctly published:

$ ros2 topic echo /clock
$ clock:
$   sec: 6
$   nanosec: 900000359
$ ---
$ clock:
$   sec: 6
$   nanosec: 916667027
$ ---
$ clock:
$   sec: 6
$   nanosec: 933333694
$ ---
$ clock:
$   sec: 6
$   nanosec: 950000362
$ ---
$ [...]

stop listening to the /clock topic with Ctrl+c.

Start the ZED ROS 2 Wrapper node with simulated data

The ZED ROS 2 Wrapper enables connection to the ZED Isaac™ Sim Extension, publishing ZED data in ROS 2 as if a real camera were present.

Prerequisites

The “ZED ROS 2 Wrapper” and the “ZED ROS 2 Examples” packages must be installed on the system.

Follow the relevant guides to install them if you have not already done so:

Launch the ZED ROS 2 node

Launch a standalone ZED ROS 2 node with simulated ZED data as input:

$ ros2 launch zed_wrapper zed_camera.launch.py camera_model:=zedx sim_mode:=true use_sim_time:=true

Launch options:

[Mandatory] sim_mode: start the ZED node in simulation mode if true.
[Mandatory] use_sim_time: force the node to wait for valid messages on the topic /clock, and use the simulation time as the reference.
[Optional] sim_address: set the address of the simulation server. Default is 127.0.0.1 and it’s valid if the node runs on the same machine as the simulator.
[Optional] sim_port: set the port of the simulation server. It must match the value of the field Streaming Port of the properties of the ZED Camera Helper Action Graph node. A different Streaming Port value for each camera is required in multi-camera simulations.

Set camera_model to match the camera placed in the simulation. The ZED Camera extension supports the ZED X stereo family (zedx, zedxm, zedxnano), with zedx being the most thoroughly tested.

You can also start a preconfigured instance of rviz2 to visualize all the information available in simulation by using the command:

$ ros2 launch zed_display_rviz2 display_zed_cam.launch.py camera_model:=zedx sim_mode:=true use_sim_time:=true

The display_zed_cam.launch.py launch file includes the zed_camera.launch.py launch file, so it provides the same parameters.

You can get the list of all the available launch parameters by using the -s launch option:

$ ros2 launch zed_wrapper zed_camera.launch.py -s
$ ros2 launch zed_display_rviz2 display_zed_cam.launch.py -s

Here’s an example of rviz2 running with the simulated information obtained by placing the ZED camera on a shelf in a simulated warehouse: