Skip to navigation
Software IntegrationIsaac Sim

SDK-free depth capture in Isaac™ Sim

Open in ClaudeOpen in ChatGPT

The ZED Camera extension can capture RGB and depth from a simulated ZED directly inside NVIDIA® Isaac™ Sim, using the simulator’s renderer, with no ZED SDK and no streaming. This is useful for quick frame/depth capture, generating datasets, or working on a machine where the ZED SDK is not installed.

For how this compares to the streaming and Isaac™ Lab tensor paths, see Choosing a data path.

The depth captured here is Isaac™ Sim’s ground-truth renderer depth (distance_to_image_plane), not the ZED SDK’s stereo-matched depth. For genuine ZED SDK depth, use the streaming path.

ZED Capture OmniGraph node

The extension provides a ZED Capture node that captures RGB, and optionally depth, each time the camera renders. Add it to an Action Graph, connect an On Playback Tick node to its ExecIn, set its properties, and press Play. The node works with stereo models only (no ZED X One).

Extension 5.2 called this node ZED Depth. Only its name in the UI changed, so existing graphs load as before.

PropertyDefaultDescription
ZED Camera Prim—The ZED camera prim in the stage (e.g. /World/ZED_X).
Camera ModelZED_XSimulated stereo model: ZED_X, ZED_XM, ZED_X_Nano, ZED_M, or ZED_2i. Detected automatically from the connected ZED Camera Prim when possible.
Lens TypeWideWide or Narrow (options depend on the model). Detected automatically like the Camera Model.
ResolutionSVGARender resolution: HD2K, HD1200, HD1080, HD720, SVGA, or VGA (filtered per model). A resolution the model does not support falls back to a supported one, with a console message.
Enable SaveOffSave the images (PNG) and depth (EXR, see Output) to disk each frame.
Output Directory—Directory to save frames. Required when Enable Save is on.
Save DepthOnWrite the depth map. Turn it off to record left and right images only, which saves most of the disk writes.

The images get the ZED Sim2Real camera model, without its auto exposure and motion blur. The node has no switch for it: to capture plain sRGB images, use ZEDDepthCamera in your own script (see Standalone scripts) with sensor_noise, vignetting, camera_sharpness and auto_white_balance set to False.

Frames are saved in the background, so saving does not hold up the simulation. If the disk or CPU cannot keep up, the app slows down rather than dropping frames, and Stop waits for the last frames to be written.

Standalone scripts

The extension also ships two capture scripts under exts/sl.sensor.camera/examples/isaac_sim/, both driven by the sl.sensor.camera.zed_depth.ZEDDepthCamera helper.

Standalone / headless (zed_depth_standalone.py)

Captures left RGB + depth for N frames, prints per-frame stats, and optionally saves PNG/EXR. It runs with no arguments: when no scene or camera is given, it builds a minimal ground + dome-light scene and spawns a ZED camera at a small default pose, so you get a usable depth capture out of the box.

The script is launched with the Python interpreter shipped with Isaac™ Sim (python.sh / python.bat, in the _build/linux-x86_64/release folder of your Isaac™ Sim build). That interpreter and the example live in two different repositories, so run python.sh from the Isaac™ Sim build directory and give the script an absolute path into your zed-isaac-sim clone:

# Linux, zero-config: minimal scene + auto-spawned ZED_X
cd <isaacsim>/_build/linux-x86_64/release
./python.sh <your_workspace>/zed-isaac-sim/exts/sl.sensor.camera/examples/isaac_sim/zed_depth_standalone.py
# Full control: load your own scene and/or use a camera already placed in it
./python.sh <your_workspace>/zed-isaac-sim/exts/sl.sensor.camera/examples/isaac_sim/zed_depth_standalone.py \
    --usd_path /path/to/scene.usd \
    --camera_prim /World/ZED_X \
    --camera_model ZED_X \
    --resolution HD1200 \
    --num_frames 10 \
    --output_dir /tmp/zed_depth
# Windows: use python.bat. Add --headless for no GUI.

Arguments (all optional): --usd_path (environment USD; omit to build a minimal scene), --camera_prim (an existing ZED camera prim; omit to spawn one), --camera_model, --resolution, --num_frames, --output_dir, --headless.

GUI Script Editor (zed_depth_example.py)

For interactive use inside the Isaac™ Sim GUI (extension enabled): load a scene with a ZED camera, paste the script into the Script Editor, edit the CONFIG block (CAMERA_PRIM_PATH, CAMERA_MODEL, RESOLUTION, CAPTURE_INTERVAL, SAVE_DIR), execute it, then press Play. RGB + depth are captured every CAPTURE_INTERVAL physics steps; press Stop to clean up.

Output

  • Console: per-capture shape and depth-range stats.
  • Files (when a save path is set): rgb_left_NNNNNN.png (left RGB) and depth_NNNNNN.exr (float32 depth in meters, 0 = invalid, losslessly compressed). For stereo models, rgb_right_NNNNNN.png (right RGB) is saved too; depth stays left-referenced.

When OpenCV is missing or cannot write EXR, depth is saved as depth_NNNNNN.npy (the same float32 array) and the console warns Depth saved as .npy instead of .exr. If the warning says the EXR writer is unavailable, set OPENCV_IO_ENABLE_OPENEXR=1 in the environment that launches Isaac™ Sim, whether you start the GUI (isaac-sim.sh / isaac-sim.bat) or a script (python.sh / python.bat):

# Linux
OPENCV_IO_ENABLE_OPENEXR=1 ./isaac-sim.sh
OPENCV_IO_ENABLE_OPENEXR=1 ./python.sh <...>/zed_depth_standalone.py --output_dir /tmp/zed_depth
# Windows (PowerShell; in cmd.exe use: set OPENCV_IO_ENABLE_OPENEXR=1)
$env:OPENCV_IO_ENABLE_OPENEXR = "1"
.\isaac-sim.bat
.\python.bat <...>\zed_depth_standalone.py --output_dir C:\zed_depth

To send the frames somewhere other than numbered PNGs, pass a frame_writer to ZEDDepthCamera: it decides where each captured frame goes, what it is called and whether it is written. Only the default writer saves in the background: a custom writer’s write() runs on the calling thread. See sl.sensor.camera.frames.FrameWriter.

See also