Software IntegrationIsaac Sim

Setting up ZED Camera in Isaac™ Sim

Open in ClaudeOpen in ChatGPT

This tutorial guides you through setting up the ZED X camera in NVIDIA® Isaac™ Sim and connecting to the ZED SDK. If you’re not familiar with Isaac™ Sim, make sure to go through the Isaac™ Sim Official documentation.

Setting up your virtual camera

Add the model to the stage

The ZED X model is available in the Isaac Assets tab on the bottom panel.

If you cannot find the ZED X Model, then you can manually add it by cloning and building the GitHub repository:

$cd <your_workspace>
$git clone git@github.com:stereolabs/zed-isaac-sim.git
$cd zed-isaac-sim

Linux:

$./build.sh

Windows:

$build.bat

image

The following camera models are available: ZED_X, ZED_X_4MM, ZED_XM, ZED_XM_4MM, ZED_XONE_UHD, ZED_XONE_GS, ZED_XONE_GS_4MM.

Select the desired model USD and drag it in the Viewport or in the Stage view.

You will notice the prim has a base_link XForm, this is the reference frame of the mounting point on the bottom of the camera.

Activate the ZED Camera extension

The simulated camera takes advantage of the ZED SDK streaming input to stream its data into the ZED SDK.
The first step is to enable the ZED Camera extension to get access to the ZED Camera Helper Omnigraph node.
In the top menu bar, navigate to Window -> Extensions to show the extension manager.

If you cannot find the ZED Camera Extension, then you can manually add it by cloning and building the GitHub repository, if you have not yet done this before:

$cd <your_workspace>
$git clone git@github.com:stereolabs/zed-isaac-sim.git
$cd zed-isaac-sim

Linux:

$./build.sh

Windows:

$build.bat

In the Extensions manager, click the menu button (hamburger icon near the search field), open Settings, and click the + button to add the path to the exts folder of the cloned repository (e.g. <your_workspace>/zed-isaac-sim/exts).

image

Enable the ZED Camera extension in the Third-Party tab.

image

Build the action graph

To initiate the virtual ZED camera streaming, we will be using the ZED Camera Helper Omnigraph node. To stream data at each simulation tick, drag and drop this node into the graph (you can type zed in the search panel to quickly find it), add its ZED Camera prim target to the camera in the scene, and connect its ExecIn input to the Tick output of the On Playback Tick node.

Starting from version 4.0.0 of the extension, two nodes are available by the ZED Camera Extension, ZED Camera Helper and ZED Bridge. Please make sure to choose the ZED Camera Helper node.

image

image

You can also configure the streamer properties in the Property panel on the right:

  • ZED Camera Prim: this parameter should be set to the ZED camera prim in the stage as shown in the picture above.
  • Streaming Bitrate: Bitrate used for the RTSP stream (default: 8000).
  • Camera Model: Defines the camera model simulated. Must match the ZED Camera Prim USD to provide accurate depth.
  • Streaming Chunk Size: Size of the streaming chunks (default: 4096).
  • FPS: Target frame rate streamed to the ZED SDK.
  • Resolution: Defines the camera resolution used. Lowering the resolution may provide better runtime performance.
  • Streaming Port: Defines which port is used for streaming. It must be an even number. A different port is required for each camera in multi-camera simulations.
  • Transport layer mode: Defines the transport mode used to stream data to the ZED SDK. Available values: IPC (inter-process communication, Linux only, same machine), NETWORK (network streaming), or BOTH. IPC provides the best performance on Linux. On Windows, only NETWORK is available.

And that’s it, you can now start the simulation to stream your stage into the ZED SDK!

To stream a monocular camera (ZED X One cameras), use the ZED One Camera Helper Node instead: Note : the serial number input is only used for Virtual stereo cameras.

image

Using IPC

It is possible to stream images to the ZED SDK using IPC instead of NETWORK streaming for better performance. This feature is only available on Linux and only when streaming to the same machine.

To use IPC, set the Transport layer mode property of the ZED Camera Helper node to IPC or BOTH.

image

Important:

To receive a stream using IPC, the IP address must be set to 127.0.0.1 or localhost.

Create a Virtual stereo Camera

It is also possible to create what we call “Virtual stereo cameras” by pairing two ZED X One cameras together. In order to do that, a calibration step is required.

  • First, open the Extension windows and enable the ZED Calibration Exporter extension in the Third-Party tab.
  • In the top Menu bar, Click ZED -> ZED Calibration Exporter. A new window will open.
  • Set the left and right cameras by selecting each prim in the Stage and click Select.
  • Select the Camera model.
  • Change the Serial Number if necessary.
  • Click on Generate. A calibration file (.conf) will be generated and saved on your machine.
  • Then, in your action graph, add a ZED Camera One Helper and set the left and right cameras.
  • You also need to set the serial number chosen during the calibration process.

Test it in ZED DepthViewer

Once streaming is started, you can test it by connecting the ZED DepthViewer to the virtual camera. Simply launch the tool, click on the streaming input button, set the IP Address (the streaming computer) and the port (set in Streaming port), and you should see the depth data coming from the virtual environment.

image