Setting up the ZED in Isaac™ Sim

This tutorial guides you through setting up a ZED camera in NVIDIA® Isaac™ Sim and connecting it to the ZED SDK. If you’re not familiar with Isaac™ Sim, make sure to go through the official Isaac™ Sim documentation.

This tutorial assumes the ZED Camera extension is already installed and enabled. If not, follow the installation steps on the Get Started page first.

Add a camera to the scene

The camera is defined by two properties: a Camera Model (base model) and a Lens Type. The lens types are Wide, Narrow (the 4.6 mm lens) and Fisheye.

Supported camera models

Type	Camera Model	Lens Type options
Stereo	`ZED_X`	Wide, Narrow
Stereo	`ZED_XM`	Wide, Narrow
Stereo	`ZED_X_Nano`	Wide
Stereo	`ZED_M` (ZED Mini)	Wide
Stereo	`ZED_2i`	Wide, Narrow
Mono	`ZED_XONE_UHD`	Wide
Mono	`ZED_XONE_GS`	Wide, Narrow
Mono	`ZED_XONE_S`	Wide, Narrow, Fisheye

Mono cameras (ZED X One) can also be paired to create virtual stereo cameras.

The extension ships USD models for all the camera bodies in the exts/sl.sensor.camera/data/usd folder. The Narrow and Fisheye lenses are variants of their model’s USD: select the desired lens with the Lens dropdown on the camera prim in the Property panel (e.g. ZED_X_Nano offers only Wide; Fisheye is only available on ZED_XONE_S). The ZED Camera Helper and ZED Camera One Helper nodes pick up that choice automatically.

Select the desired model USD and drag it into the Viewport or the Stage view.

You will notice that the prim has a base_link XForm; this is the reference frame of the mounting point on the bottom of the camera.

Coming from extension 5.2: the ZED X One UHD model now has its real housing, so its camera sits 9.3 mm further back and 3.8 mm higher relative to the asset’s origin, and the ZED Mini model is no longer upside down. Re-check mounts tuned to the old models, and remove any rotation that compensated for the upside-down ZED Mini.

ZED Cameras panel

The ZED Cameras panel lists every ZED camera in the stage and lets you switch lenses without opening the Property panel. It is docked next to the Content browser and opens automatically when Isaac™ Sim starts. If you close it, reopen it from Stereolabs → ZED Cameras.

Each row shows a camera’s thumbnail, prim name, and model, tagged stereo or mono. To work with a camera:

Switch its lens: click one of the Wide / Narrow / Fisheye buttons. Only the lenses the model supports are shown, and the current lens is highlighted. Fixed-lens models (ZED Mini, ZED X One UHD, ZED X Nano) display their single lens as a plain tag.
Select it in the stage: click anywhere on its row. The selected camera’s row is outlined.

Switching the lens here edits the same Lens variant as the camera prim’s dropdown, so the viewport, the model the ZED Camera Helper detects, and the streamed calibration all stay in sync.

Build the Action Graph

The simulated camera streams to the ZED SDK through its streaming input.

To initiate the simulated ZED camera streaming, we will use the ZED Camera Helper OmniGraph node. To stream data at each simulation tick, drag and drop this node into the graph (you can type zed in the search panel to quickly find it), set its ZED Camera prim target to the camera in the scene, and connect its ExecIn input to the Tick output of the On Playback Tick node.

The ZED Camera extension also includes a lower-level ZED Stream node. For most use cases, make sure to use the ZED Camera Helper node (or the ZED Camera One Helper node for monocular ZED X One cameras).

You can also configure the ZED Camera Helper from the Property panel on the right:

Property	Default	Description
ZED Camera Prim	—	The ZED camera prim to stream, as shown in the picture above. The target picker only lists ZED camera prims.
Camera Model / Lens Type	Auto-detected	Read from the connected ZED Camera Prim (its name and Lens selection) and shown as a read-only summary. If no prim is connected, editable dropdowns are shown as a fallback.
Resolution	Model-dependent	Streaming resolution, filtered to the values valid for the selected model: `HD1200`, `HD1080`, `SVGA` (ZED X family, ZED X One GS and ZED X One S); `HD2K`, `HD1080`, `HD720`, `VGA` (ZED Mini and ZED 2i); `HD4K`, `QHDPLUS`, `HD1200`, `HD1080` (ZED X One UHD). Lowering it can improve runtime performance.
FPS	`30`	Frame rate streamed to the ZED SDK. The camera renders only at this rate rather than on every simulation frame, so lowering it reduces GPU cost, especially in multi-camera scenes.
Transport layer mode	`BOTH`	How data is streamed to the ZED SDK: `IPC` (same machine), `NETWORK` (network streaming), or `BOTH`. See Using IPC.
Streaming Port	`30000`	Port used for streaming. Must be an even number, and unique per camera in multi-camera simulations.
Streaming Bitrate	`8000`	Bitrate (Kbps) of the network stream. Used only when not streaming over IPC.
Streaming Chunk Size	`4096`	Size (bytes) of the network streaming chunks. Used only when not streaming over IPC.
Stream Depth	Off	Streams the left image and a depth map instead of the stereo pair. See Streaming depth.
Apply ZED Sim2Real	On	Master switch of the ZED Sim2Real camera model, a model of the real ZED X. Off streams the render as it is. Editable live.
Auto exposure, Auto white balance, Sensor noise, Motion blur, Vignetting, Camera sharpness	On	The camera model’s effects, each on its own switch. They need Apply ZED Sim2Real. Editable live.

Press Play to start streaming your stage into the ZED SDK.

A simulated ZED streaming live from Isaac™ Sim into ZED Depth Viewer, with the stereo pair and the ZED SDK’s point cloud.

To stream a monocular camera (ZED X One), use the ZED Camera One Helper node instead.

On the ZED Camera One Helper, the Serial Number input is only used for virtual stereo cameras (both camera prims set). Set it to the serial number you exported with the Calibration Exporter.

Simulated camera serial numbers

Simulated cameras use a short list of reserved serial numbers. The receiver needs internet access the first time it opens a camera, to download its calibration file. There are only a few of these serial numbers per model and lens, so only a few cameras of the same model and lens stream at once.

Virtual stereo cameras (two ZED X One) use the serial number and calibration file from the Calibration Exporter instead. That file is not downloaded: when the receiver runs on another machine, copy it there.

Streaming depth

By default, the ZED Camera Helper streams the left + right stereo pair. You can instead stream the left image together with a depth map directly into the ZED SDK: enable the Stream Depth property on the node.

The depth is Isaac™ Sim’s ground-truth renderer depth, delivered to the ZED SDK through its ingestCustomDepth feature (requires ZED SDK ≥ 5.4.0). It is useful for feeding a known-good depth map into a ZED SDK pipeline for testing. The streamed depth is 896 pixels wide, with the image’s aspect ratio: 896×560 at HD1200 and SVGA, 896×504 at the 16:9 resolutions.

On the ZED Camera One Helper node, Stream Depth requires a virtual stereo pair (both camera prims set): a single mono camera cannot produce depth on real hardware, so the option is ignored.

ZED Sim2Real camera model

Simulated images are too clean. ZED Sim2Real makes them look like they came from a real ZED X:

Auto exposure: exposure time and gain adapt to the scene like a real ZED X’s, from direct sun to dim rooms.
Auto white balance, with the real camera’s limits.
Sensor noise that grows with the gain, so dim scenes are noisier.
Motion blur over each frame’s exposure time, so longer exposures blur more.
Lens vignetting and the camera’s sharpness.

The same ZED X model is applied to every camera model, including the ZED Mini, ZED 2i and ZED X One.

Left: real ZED X. Middle: raw Isaac™ Sim render. Right: the same render with the ZED Sim2Real model applied.

Apply ZED Sim2Real, on by default, is the master switch. Each effect also has its own switch under it, in the ZED Sim2Real group of the ZED Camera Helper and ZED Camera One Helper nodes. All of them can be changed live while the timeline is playing. With Apply ZED Sim2Real off, the render streams as it is: the scene’s own camera exposure and plain sRGB.

The same model runs on the ZED Capture node, without auto exposure and motion blur, and in Isaac™ Lab, without motion blur.

If the ZED Sim2Real library is missing from your build, images are streamed without the camera model and the console logs a warning.

Coming from extension 5.2: images now have a real ZED X’s color response instead of the renderer’s tone mapping, so scenes tuned on 5.2 look brighter in the mid-tones and darker in the shadows. The 5.2 Sim2Real settings (Scene Lux and the tuning controls) are gone: graphs saved with 5.2 drop them on load and get Apply ZED Sim2Real switched on, even where it was off.

Scene lighting

With auto exposure on, a scene lit far too dimly or too brightly gives the dark, noisy or blown-out image a real ZED X would. The console then warns that the scene is too dark for a real ZED X (or too bright), with its estimated illuminance. This usually means the lights use the wrong units:

a DistantLight of intensity I gives about 3.1 × I lux;
DomeLight and SphereLight intensities are luminance, in nits;
an indoor room is 200 to 800 lux;
direct sunlight is about 100,000 lux.

Using IPC

On the same machine, IPC streams to the ZED SDK faster than network streaming. It is available on Linux and Windows (since extension v5.2.0).

To use IPC, set the Transport layer mode property of the ZED Camera Helper node to IPC or BOTH.

To receive a stream using IPC, the IP address must be set to 127.0.0.1 or localhost.

Multi-camera streaming

You can stream several simulated ZED cameras simultaneously: several cameras on a single robot (for example a front ZED X plus ZED X One side cameras), or several robots each carrying their own. Stereo and mono models can be mixed freely in the same scene.

Add one ZED Camera Helper node per camera (ZED Camera One Helper for mono cameras); a single Action Graph can drive them all.
Give each camera a unique even Streaming Port (30000, 30002, 30004, …). Reusing a port across two cameras is an error. Each camera arrives as an independent stream, with its own serial number and calibration, so the ZED SDK or the ZED ROS 2 Wrapper sees them exactly like separate physical cameras.
Only a few cameras of the same model and lens stream at once, see Simulated camera serial numbers.
In large scenes, lower the FPS or Resolution of individual cameras to reduce GPU cost.

If you use ZED Fusion, the fusion_calibration example generates the Fusion calibration file directly from the camera prim poses in your stage. Only four ZED X with the same lens stream at once: with more, give some of them the narrow lens.

Create a Virtual Stereo Camera

You can also create virtual stereo cameras by pairing two ZED X One cameras together, which requires a calibration step.

First, open the Extensions window and enable the ZED Calibration Exporter extension in the Third-Party tab.
In the top menu bar, click Stereolabs -> ZED Calibration Exporter. A new window will open.
Set the left and right cameras by selecting each prim in the Stage and clicking Select.
Select the Camera model and the Lens type. The lens list only shows the lenses available for the selected model.
Change the Serial Number if necessary. It must start with 11 (110000001 to 119999999).
Click Generate. The calibration file SN<serial>.conf is written to the ZED SDK settings folder (C:/ProgramData/Stereolabs/settings/ on Windows, /usr/local/zed/settings/ on Linux) of the Isaac™ Sim machine. When the ZED SDK receives the stream on another machine, copy the file to the settings folder there too.

Then, in your Action Graph, add a ZED Camera One Helper and set the left and right cameras.
You also need to set the serial number chosen during the calibration process.

Calibrations exported before extension 5.3.0 are wrong for any rig whose two cameras are not side by side and parallel (toe-in, tilt, or a vertical or depth offset). Re-export the calibration of such rigs. A side-by-side, parallel pair is unaffected.

Test it in ZED Depth Viewer

Once streaming is started, you can test it by connecting ZED Depth Viewer to the simulated camera. Launch the tool, click on the streaming input button, set the IP Address (the streaming computer) and the port (set in Streaming port), and you should see the depth data coming from the virtual environment.

Troubleshooting

Stream not detected. Confirm the simulation is playing, the Streaming Port is even and unique, and, for network streaming, that the port is open in your firewall. For IPC, the receiver’s IP must be 127.0.0.1 or localhost.
Maximum number of ... camera reached in the console. All the reserved serial numbers of that model and lens are in use: stream fewer cameras of that model and lens at once.
Streamer initialization failed in the console. The message names the cause and what to do about it. After fixing it, Stop and Play again: the camera does not retry on its own.
- PORT_UNAVAILABLE: the port is already in use. Stop the other stream or pick another port.
- INVALID_PORT: the port must be even.
- ENCODER_LOAD_FAILED, ENCODER_CREATION_FAILED: the hardware video encoder is unavailable or rejected the configuration. Check the GPU driver, the resolution and the FPS.
- INVALID_SERIAL_NUMBER: the serial number is not valid. For a virtual stereo camera, use the one exported with the Calibration Exporter (110000001 to 119999999).
- INVALID_INPUT: among others, a resolution the camera model does not support.
Not streaming: the IMU at ... gives no reading in the console. The camera’s Imu_Sensor needs its Enabled attribute on, and an enabled rigid body on the camera or a parent prim. To hold a camera still, tick Kinematic Enabled instead of unticking Rigid Body Enabled. Then Stop and Play.
The Calibration Exporter reports an error. It refuses serial numbers outside 110000001 to 119999999, and reports a calibration file it cannot write: check that you can write to the ZED SDK settings folder (/usr/local/zed/settings/ on Linux).
One camera stops, the others lose their IPC stream. On IPC or BOTH, closing one camera’s stream also interrupts IPC delivery for the other cameras in the scene. Their network streams are unaffected.
The image is black, noisy or blown out. The scene is too dark or too bright for a real ZED X. See Scene lighting.
First frame is slow or seems to hang. The first rendered frame compiles the RTX shaders and can take minutes on a cold cache, with no progress shown. Later runs start fast.