Skip to navigation

Setting up the ZED in Isaac™ Sim

Open in ClaudeOpen in ChatGPT

This tutorial guides you through setting up a ZED camera in NVIDIA® Isaac™ Sim and connecting it to the ZED SDK. If you’re not familiar with Isaac™ Sim, make sure to go through the official Isaac™ Sim documentation.

This tutorial assumes the ZED Camera extension is already installed and enabled. If not, follow the installation steps on the Get Started page first.

Add a camera to the scene

The camera is defined by two properties: a Camera Model (base model) and a Lens Type. The lens types are Wide, Narrow (the 4.6 mm lens) and Fisheye.

Supported camera models

TypeCamera ModelLens Type options
StereoZED_XWide, Narrow
StereoZED_XMWide, Narrow
StereoZED_X_NanoWide
StereoZED_M (ZED Mini)Wide
StereoZED_2iWide, Narrow
MonoZED_XONE_UHDWide
MonoZED_XONE_GSWide, Narrow
MonoZED_XONE_SWide, Narrow, Fisheye

Mono cameras (ZED X One) can also be paired to create virtual stereo cameras.

The extension ships USD models for all the camera bodies in the exts/sl.sensor.camera/data/usd folder. The Narrow and Fisheye lenses are variants of their model’s USD: select the desired lens with the Lens dropdown on the camera prim in the Property panel (e.g. ZED_X_Nano offers only Wide; Fisheye is only available on ZED_XONE_S). The ZED Camera Helper and ZED Camera One Helper nodes pick up that choice automatically.

Select the desired model USD and drag it into the Viewport or the Stage view.

You will notice that the prim has a base_link XForm; this is the reference frame of the mounting point on the bottom of the camera.

Coming from extension 5.2: the ZED X One UHD model now has its real housing, so its camera sits 9.3 mm further back and 3.8 mm higher relative to the asset’s origin, and the ZED Mini model is no longer upside down. Re-check mounts tuned to the old models, and remove any rotation that compensated for the upside-down ZED Mini.

ZED Cameras panel

The ZED Cameras panel lists every ZED camera in the stage and lets you switch lenses without opening the Property panel. It is docked next to the Content browser and opens automatically when Isaac™ Sim starts. If you close it, reopen it from Stereolabs → ZED Cameras.

image

Each row shows a camera’s thumbnail, prim name, and model, tagged stereo or mono. To work with a camera:

  • Switch its lens: click one of the Wide / Narrow / Fisheye buttons. Only the lenses the model supports are shown, and the current lens is highlighted. Fixed-lens models (ZED Mini, ZED X One UHD, ZED X Nano) display their single lens as a plain tag.
  • Select it in the stage: click anywhere on its row. The selected camera’s row is outlined.

Switching the lens here edits the same Lens variant as the camera prim’s dropdown, so the viewport, the model the ZED Camera Helper detects, and the streamed calibration all stay in sync.

The panel’s Settings tab holds two settings shared by every camera:

Build the Action Graph

The simulated camera streams to the ZED SDK through its streaming input.

To initiate the simulated ZED camera streaming, we will use the ZED Camera Helper OmniGraph node. To stream data at each simulation tick, drag and drop this node into the graph (you can type zed in the search panel to quickly find it), set its ZED Camera prim target to the camera in the scene, and connect its ExecIn input to the Tick output of the On Playback Tick node.

The ZED Camera extension also includes a lower-level ZED Stream node. For most use cases, make sure to use the ZED Camera Helper node (or the ZED Camera One Helper node for monocular ZED X One cameras).

image image

You can also configure the ZED Camera Helper from the Property panel on the right:

image
PropertyDefaultDescription
ZED Camera PrimNoneThe ZED camera prim to stream, as shown in the picture above. The target picker only lists ZED camera prims.
Camera Model / Lens TypeAuto-detectedRead from the connected ZED Camera Prim (its name and Lens selection) and shown as a read-only summary. If no prim is connected, editable dropdowns are shown as a fallback.
ResolutionModel-dependentStreaming resolution, filtered to the values valid for the selected model: HD1200, HD1080, SVGA (ZED X family, ZED X One GS and ZED X One S); HD2K, HD1080, HD720, VGA (ZED Mini and ZED 2i); HD4K, QHDPLUS, HD1200, HD1080 (ZED X One UHD). Lowering it can improve runtime performance.
Camera Index-1Picks a specific simulated camera, with its own serial number and calibration. -1 takes the first free one when you press Play. Not used in legacy mode.
FPS30Frame rate streamed to the ZED SDK: 15, 30, 60 or 120. The camera renders only at this rate rather than on every simulation frame, so lowering it reduces GPU cost, especially in multi-camera scenes. Play can change the app’s timing so that every frame is sent: see Stream frame rate and app timing.
Transport layer modeBOTHHow data is streamed to the ZED SDK: IPC (same machine), NETWORK (network streaming), or BOTH. See Using IPC.
Streaming Port30000Port used for streaming. Must be an even number. Play moves a camera whose port another camera already uses, see Multi-camera streaming.
Streaming Bitrate8000Bitrate (Kbps) of the network stream. Used only when not streaming over IPC.
Streaming Chunk Size4096Size (bytes) of the network streaming chunks. Used only when not streaming over IPC.
Stream DepthOffStreams the left image and a depth map instead of the stereo pair. See Streaming depth.
Apply ZED Sim2RealOnMaster switch of the ZED Sim2Real camera model, a model of the real ZED X. Off streams the render as it is. Editable live.
Auto exposure, Auto white balance, Sensor noise, Motion blur, Vignetting, Camera sharpnessOnThe camera model’s effects, each on its own switch. They need Apply ZED Sim2Real. Editable live.
Focus blurOffThe lens’s fixed-focus blur on close surfaces. Needs Apply ZED Sim2Real. Editable live. See Focus blur.

Press Play to start streaming your stage into the ZED SDK.

A simulated ZED streaming live from Isaac™ Sim into ZED Depth Viewer, with the stereo pair and the ZED SDK’s point cloud.

To stream a monocular camera (ZED X One), use the ZED Camera One Helper node instead.

On the ZED Camera One Helper, the Serial Number input is only used for virtual stereo cameras (both camera prims set). Set it to the serial number you exported with the Calibration Exporter.

image

Simulated camera serial numbers and calibration

By default, the ZED SDK generates each simulated camera: it has its own serial number and a calibration slightly different from the next camera’s, like real units, and its images are rendered with that calibration. The receiver regenerates the calibration from the serial number and rectifies the images as it does a real camera’s. There is no limit on the number of cameras, nothing to download, and it works offline.

  • Picking a camera. The same model, lens and Camera Index always give the same serial number and calibration, from one run to the next. The default, -1, takes the first free index at Play, so set the index explicitly if a camera must keep its serial number when cameras are added or removed. The serial number is printed in the console when streaming starts.
  • Right eye. While the simulation runs, the right eye moves to its calibrated pose. The scene file keeps the asset’s pose.
  • Receiver. Needs ZED SDK 5.6.0 or newer. Disable self-calibration on the receiver (InitParameters::camera_disable_self_calib): the simulated calibration is exact, and self-calibration would only move it away. Recordings of a simulated camera embed its calibration.

The ZED Capture node and Isaac™ Lab use the same cameras.

Legacy mode

For a receiver older than ZED SDK 5.6.0, switch the Camera mode to Legacy serial number in the Settings tab of the ZED Cameras panel. Without the UI, set the /persistent/exts/sl.sensor.camera/simCameraMode setting to "legacy". The mode applies to every camera, from the next Play.

Simulated cameras then use a short list of reserved serial numbers, as in extension 5.3. The receiver needs internet access the first time it opens a camera, to download its calibration file. This works with any receiver from ZED SDK 5.5.0, but allows only a few cameras per model and lens, all with the same ideal calibration, and Camera Index is not used. A stream keeps its serial number from one run to the next as long as its port does not change.

The cameras also fall back to this mode, with a console warning, when the extension’s ZED library cannot generate simulated cameras.

Virtual stereo cameras (two ZED X One) always use the serial number and calibration file from the Calibration Exporter, in both modes. That file is not downloaded: when the receiver runs on another machine, copy it there.

Streaming depth

By default, the ZED Camera Helper streams the left + right stereo pair. You can instead stream the left image together with a depth map directly into the ZED SDK: enable the Stream Depth property on the node.

The depth is Isaac™ Sim’s ground-truth renderer depth, delivered to the ZED SDK through its ingestCustomDepth feature (requires ZED SDK ≥ 5.4.0). It is useful for feeding a known-good depth map into a ZED SDK pipeline for testing. The streamed depth is 896 pixels wide, with the image’s aspect ratio: 896×560 at HD1200 and SVGA, 896×504 at the 16:9 resolutions.

On the ZED Camera One Helper node, Stream Depth requires a virtual stereo pair (both camera prims set): a single mono camera cannot produce depth on real hardware, so the option is ignored.

ZED Sim2Real camera model

Simulated images are too clean. ZED Sim2Real makes them look like they came from a real ZED X:

  • Auto exposure: exposure time and gain adapt to the scene like a real ZED X’s, from direct sun to dim rooms.
  • Auto white balance, with the real camera’s limits.
  • Sensor noise that grows with the gain, so dim scenes are noisier.
  • Motion blur over each frame’s exposure time, so longer exposures blur more.
  • Lens vignetting and the camera’s sharpness.
  • Focus blur (off by default): the lens’s fixed focus, with close surfaces blurred by their distance.

The auto exposure and auto white balance meter the right eye of a stereo camera, as a real ZED X does. Mono cameras and left + depth streams meter the left eye.

The same ZED X model is applied to every camera model, including the ZED Mini, ZED 2i and ZED X One.

ZED Sim2Real comparison

Left: real ZED X. Middle: raw Isaac™ Sim render. Right: the same render with the ZED Sim2Real model applied.

Apply ZED Sim2Real, on by default, is the master switch. Each effect also has its own switch under it, in the ZED Sim2Real group of the ZED Camera Helper and ZED Camera One Helper nodes. All of them can be changed live while the timeline is playing. With Apply ZED Sim2Real off, the render streams as it is: the scene’s own camera exposure and plain sRGB.

The same model runs on the ZED Capture node, with every effect, and in Isaac™ Lab, without motion blur or focus blur.

ZED camera images render with DLSS Quality whatever the application’s DLSS mode, since Performance leaves stair-stepped edges; the viewport keeps its own mode. The /exts/sl.sensor.camera/renderDlssMode setting picks another mode (performance, balanced, quality, auto), or "" to leave it to the application.

If the ZED Sim2Real library is missing from your build, images are streamed without the camera model and the console logs a warning.

Coming from extension 5.2: images now have a real ZED X’s color response instead of the renderer’s tone mapping, so scenes tuned on 5.2 look brighter in the mid-tones and darker in the shadows. The 5.2 Sim2Real settings (Scene Lux and the tuning controls) are gone: graphs saved with 5.2 drop them on load and get Apply ZED Sim2Real switched on, even where it was off.

Scene lighting

With auto exposure on, a scene lit far too dimly or too brightly gives the dark, noisy or blown-out image a real ZED X would. The console then warns that the scene is too dark for a real ZED X (or too bright), with its estimated illuminance. This usually means the lights use the wrong units:

  • a DistantLight of intensity I gives about 3.1 × I lux;
  • DomeLight and SphereLight intensities are luminance, in nits;
  • an indoor room is 200 to 800 lux;
  • direct sunlight is about 100,000 lux.

Focus blur

The ZED X lenses have a fixed focus: the wide lens is focused at 1.15 m, the narrow lens at 2 m. A real camera blurs what is closer than that, most visibly with the narrow lens at manipulation distances. Focus blur adds it to the ZED X, ZED X Mini, ZED X One GS and ZED X One S.

It is off by default because each eye also renders a depth image while it is on: about +5% frame time on a close-up scene at HD1200 on an RTX 5090. The /exts/sl.sensor.camera/focusBlurFNumber and /exts/sl.sensor.camera/focusBlurDistance (meters) settings override the lenses’ figures for every camera, for example for a unit focused nearer.

Light flicker

Mains-powered lamps make a real camera’s frames flicker: a 10 Hz beat at 30 fps under 50 Hz mains, weaker with longer exposures. It is off by default, and applies to streams and SDK-free captures alike. To add it, set /exts/sl.sensor.camera/lightFlickerStd to office_led (LED panels), lamps (halogen, fluorescent or LED lamps) or a fraction such as 0.01, and /exts/sl.sensor.camera/lightFlickerMainsHz to 60 where the mains is 60 Hz.

Exposure control from the receiver

A ZED SDK application receiving the stream can control the exposure with Camera::setCameraSettings(), as on a real camera: AEC_AGC, EXPOSURE_TIME, ANALOG_GAIN, DIGITAL_GAIN, EXPOSURE_COMPENSATION and the AUTO_*_RANGE settings. It can also read each frame’s exposure time and gains, which are kept in its SVO recordings.

This needs Auto exposure on. setCameraSettings() returns ERROR_CODE::INVALID_FUNCTION_PARAMETERS for the settings the simulated camera does not apply. A receiver older than ZED SDK 5.5.1 reports those settings as set, and its SVO recordings of the stream as received (not transcoded) lose the exposure.

Using IPC

On the same machine, IPC streams to the ZED SDK faster than network streaming. It is available on Linux and Windows (since extension v5.2.0).

To use IPC, set the Transport layer mode property of the ZED Camera Helper node to IPC or BOTH.

image

To receive a stream using IPC, the IP address must be set to 127.0.0.1 or localhost.

Stream frame rate and app timing

A stream sends at most one frame per app update, so it sends every frame only when no app update spans more than one frame period of simulation time. Isaac™ Sim’s default timing, 60 Hz updates and physics, fits 15, 30 and 60 fps streams.

When a stream does not fit, Play can change the app’s update rate, and the physics rate when needed. It says so in the console, and Stop puts them back. Stream timing, in the Settings tab of the ZED Cameras panel, picks how, for every camera:

Stream timingSettingApp update rate at Play, for a 30 fps stream
Fit the stream fps (default)fitA timing that fits is kept. Otherwise the highest multiple of the stream fps at or below the app’s update rate, or the stream fps when the app updates slower than the stream.
One update per frameframeThe stream fps: 60 Hz and 50 Hz both become 30 Hz.
Keep the app’s timingoffUnchanged.

Scripts set /persistent/exts/sl.sensor.camera/streamTiming instead. The choice applies from the next Play.

  • When the update rate changes, the physics rate goes up to the next multiple of it, never down: 200 Hz becomes 210 Hz with 30 Hz updates. A new physics rate changes anything tuned to it, such as a robot’s control policy.
  • A new update rate changes the speed of USD time-sampled animation: animation authored at 60 time codes per second plays at half speed with 30 Hz updates.
  • Use One update per frame when the machine cannot keep up with the app’s update rate in real time.
  • With Keep the app’s timing and a timing that does not fit, a stream sends every second frame or fewer, evenly spaced, and declares that rate to the receiver: 15 fps for a 30 fps stream over 15 or 20 Hz app updates. The console warns once per camera.
  • The app is not retimed for a camera added during Play, nor when Play resumes a pause: Stop, then Play.

The extension reference has the full rules.

Multi-camera streaming

You can stream several simulated ZED cameras simultaneously: several cameras on a single robot (for example a front ZED X plus ZED X One side cameras), or several robots each carrying their own. Stereo and mono models can be mixed freely in the same scene.

  • Add one ZED Camera Helper node per camera (ZED Camera One Helper for mono cameras); a single Action Graph can drive them all.
  • Each camera streams on its own even Streaming Port (30000, 30002, 30004, …). When two cameras are set to the same port, Play moves one of them to the next free even port and names it in a console warning: check the port the receiver must open. Each camera arrives as an independent stream, with its own serial number and calibration, so the ZED SDK or the ZED ROS 2 Wrapper sees them exactly like separate physical cameras.
  • Set each node’s Camera Index if your application identifies cameras by serial number: with the default -1, adding or removing a camera can shift the serial numbers of the others.
  • In legacy mode, only a few cameras of the same model and lens stream at once.
  • In large scenes, lower the FPS or Resolution of individual cameras to reduce GPU cost.

If you use ZED Fusion, the fusion_calibration example generates the Fusion calibration file directly from the camera prim poses in your stage. Its seven-camera scene needs the default camera mode, since legacy mode allows fewer.

Create a Virtual Stereo Camera

You can also create virtual stereo cameras by pairing two ZED X One cameras together, which requires a calibration step.

  • First, open the Extensions window and enable the Stereolabs ZED Calibration Exporter extension in the Third-Party tab.
  • In the top menu bar, click Stereolabs -> ZED Calibration Exporter. A new window will open.
  • Set the left and right cameras by selecting each prim in the Stage and clicking Select.
  • Select the Camera model and the Lens type. The lens list only shows the lenses available for the selected model.
  • Change the Serial Number if necessary. It must start with 11 (110000001 to 119999999).
  • Click Generate. The calibration file SN<serial>.conf is written to the ZED SDK settings folder (C:/ProgramData/Stereolabs/settings/ on Windows, /usr/local/zed/settings/ on Linux) of the Isaac™ Sim machine. When the ZED SDK receives the stream on another machine, copy the file to the settings folder there too.
  • Then, in your Action Graph, add a ZED Camera One Helper and set the left and right cameras.
  • You also need to set the serial number chosen during the calibration process.

Calibrations exported before extension 5.3.0 are wrong for any rig whose two cameras are not side by side and parallel (toe-in, tilt, or a vertical or depth offset). Re-export the calibration of such rigs. A side-by-side, parallel pair is unaffected.

Test it in ZED Depth Viewer

Once streaming is started, you can test it by connecting ZED Depth Viewer to the simulated camera. Launch the tool, click on the streaming input button, set the IP Address (the streaming computer) and the port (set in Streaming port), and you should see the depth data coming from the virtual environment.

image

Troubleshooting

  • Stream not detected. Confirm the simulation is playing, the Streaming Port is even and is the port the receiver opens (Play changes a port another camera already uses), and, for network streaming, that the port is open in your firewall. For IPC, the receiver’s IP must be 127.0.0.1 or localhost.
  • Maximum number of ... camera reached in the console. In legacy mode, all the reserved serial numbers of that model and lens are in use. Switch the camera mode back to SDK virtual camera for more cameras (receiver on ZED SDK 5.6.0 or newer).
  • The receiver fails with CALIBRATION_FILE_NOT_AVAILABLE for a serial number like 42997950. Its ZED SDK is older than 5.6.0, so it cannot regenerate the calibration of a simulated camera. Update the receiver, or switch the camera mode to Legacy serial number.
  • cannot generate simulated cameras in the console. The extension’s ZED library is older than 5.6.0, so the cameras use legacy mode. Rebuild the extension to update it. create_virtual_camera failed means that model, lens and resolution cannot be simulated; Camera index ... is already used means two nodes ask for the same camera.
  • Streamer initialization failed in the console. The message names the cause and what to do about it. After fixing it, Stop and Play again: the camera does not retry on its own.
    • PORT_UNAVAILABLE: another application already uses the port. Stop it or pick another port.
    • INVALID_PORT: the port must be even.
    • ENCODER_LOAD_FAILED, ENCODER_CREATION_FAILED: the hardware video encoder is unavailable or rejected the configuration. Check the GPU driver, the resolution and the FPS.
    • INVALID_SERIAL_NUMBER: the serial number is not valid. For a virtual stereo camera, use the one exported with the Calibration Exporter (110000001 to 119999999).
    • INVALID_INPUT: among others, a resolution the camera model does not support.
  • Not streaming: the IMU at ... gives no reading in the console. The camera’s Imu_Sensor needs its Enabled attribute on, and an enabled rigid body on the camera or a parent prim. To hold a camera still, tick Kinematic Enabled instead of unticking Rigid Body Enabled. Then Stop and Play.
  • The Calibration Exporter reports an error. It refuses serial numbers outside 110000001 to 119999999, and reports a calibration file it cannot write: check that you can write to the ZED SDK settings folder (/usr/local/zed/settings/ on Linux).
  • The image is black, noisy or blown out. The scene is too dark or too bright for a real ZED X. See Scene lighting.
  • First frame is slow or seems to hang. The first rendered frame compiles the RTX shaders and can take minutes on a cold cache, with no progress shown. Later runs start fast.

See also