Skip to navigation
Software IntegrationIsaac Sim

Using the ZED in Isaac™ Lab

Open in ClaudeOpen in ChatGPT

Isaac™ Lab is NVIDIA®‘s framework for robot learning built on top of Isaac™ Sim. The ZED Camera extension lets you use a simulated ZED camera directly inside Isaac™ Lab and read its stereo RGB pair and a depth map as in-process tensors, with no ZED SDK and no network streaming.

This is the path for reinforcement learning and imitation learning workflows, where you want camera observations as tensors on the GPU rather than a ZED SDK stream. For how it compares to the streaming and SDK-free depth paths, see Choosing a data path.

The depth here is Isaac™ Sim’s simulated renderer depth (distance_to_image_plane), not the ZED SDK’s stereo-matched depth. For genuine ZED SDK depth, use the streaming path.

How it works

The ZED USD model (e.g. ZED_X.usdc) has authored CameraLeft / CameraRight prims at the real stereo baseline, and apply_zed_optics() gives them the model’s field of view and lens. In Isaac™ Lab you reference that USD, attach an isaaclab.sensors.Camera sensor to each of those prims, and read rgb and distance_to_image_plane each simulation step. To place or mount the camera, you move the whole USD root so the stereo rig moves rigidly with it.

Requirements

  • An Isaac™ Lab installation (verified on Isaac™ Lab 3.0 / Kit 110.3.0, Isaac™ Sim 6.1).
  • The ZED Camera extension 5.3.0 or newer (from the zed-isaac-sim repository) on the Python path. The shipped examples handle this automatically via a small bootstrap import.
  • Launch through the Isaac™ Lab launcher (isaaclab.bat / isaaclab.sh). The shipped examples enable cameras automatically; when launching your own scripts, pass --enable_cameras: RGB and depth are render-based, so without it the renderer is off and frames are empty.

The first rendered frame compiles RTX shaders and can take several minutes on a cold cache. Later runs are fast.

Quick start

The extension ships ready-to-run examples under exts/sl.sensor.camera/examples/isaaclab/. To add a single ZED X to a scene and read its stereo RGB + depth tensors:

# Windows
isaaclab.bat -p exts/sl.sensor.camera/examples/isaaclab/zed_single.py --resolution SVGA --save_dir C:\zed_lab_out
# Linux
./isaaclab.sh -p exts/sl.sensor.camera/examples/isaaclab/zed_single.py --resolution SVGA --save_dir /tmp/zed_lab_out

Other examples include a vectorized multi-environment capture (zed_multi_env.py) and ports of Isaac™ Lab demos (walking quadruped, Franka lift, imitation-learning HDF5 recording) with a real ZED USD model mounted on the robot: the ZED X, or the ZED X Nano on the Franka wrist.

Franka lift task with ZED observations.

Using the sensor in your own script

The sl.sensor.camera package is importable without enabling the Kit extension (a plain sys.path insert is enough), and its helpers do the ZED-specific work:

import sys
sys.path.insert(0, r"/path/to/zed-isaac-sim/exts/sl.sensor.camera")
from isaaclab.app import AppLauncher
from sl.sensor.camera.isaaclab_utils import add_zed_cli_args, add_zed_kit_args
# ... parse args (add_zed_cli_args adds --camera_model, --resolution, ...), then:
args.enable_cameras = True   # RGB and depth are render-based
add_zed_kit_args(args)       # needed for the ZED lens, before AppLauncher
app = AppLauncher(args).app
import isaaclab.sim as sim_utils
from isaaclab.sim import SimulationContext
from isaaclab.sensors import Camera
from sl.sensor.camera.isaaclab_utils import (apply_zed_optics, make_camera_cfg, unwrap_output,
                                             wait_for_camera_data)
from sl.sensor.camera.utils import get_camera_paths, get_camera_usd_path
sim = SimulationContext(sim_utils.SimulationCfg(device=args.device))
# 1) Reference the ZED USD and place the WHOLE root (the cameras inherit the pose + baseline).
root = f"/World/{args.camera_model}"
zed = sim_utils.UsdFileCfg(usd_path=get_camera_usd_path(args.camera_model))
zed.func(root, zed)
# 2) Apply the model's field of view and lens, before the Camera sensors start.
apply_zed_optics(sim.stage, root, args.camera_model, args.resolution)
# 3) Attach Camera sensors to the authored stereo prims.
paths = get_camera_paths(root, args.camera_model)   # {"left": .../CameraLeft, "right": ..., "imu": ...}
left  = Camera(make_camera_cfg(paths["left"],  args.camera_model, args.resolution,
                               data_types=["rgb", "distance_to_image_plane"]))
right = Camera(make_camera_cfg(paths["right"], args.camera_model, args.resolution, data_types=["rgb"]))
sim.reset()
dt = sim.get_physics_dt()
wait_for_camera_data(sim, [left, right])            # step until the annotators deliver data
while app.is_running():
    sim.step()
    left.update(dt); right.update(dt)
    rgb_left = unwrap_output(left.data.output, "rgb")                       # (num_cameras, H, W, 3)
    depth    = unwrap_output(left.data.output, "distance_to_image_plane")   # (num_cameras, H, W, 1), meters

When mounting the ZED on a robot, transform the USD root; never move the CameraLeft / CameraRight prims individually, or you break the stereo rig. On a moving link, mount it with the make_zed_usd_link_mount() and author_zed_link_joint() helpers, as the robot examples do.

Extension 5.3.0 removed make_stereo_camera_cfgs() and the spawn_pinhole, clipping_range and offset arguments of make_camera_cfg(). Mount the real ZED USD as above.

It also fixed author_zed_link_joint(), which misread its angles: (0, 0, 0) turned the camera by 180°. The angles are now RotateXYZ degrees. Re-tune the angles of a mount set up with an earlier version.

ZED Sim2Real camera model

To make the RGB tensors look like a real ZED X camera instead of a clean render, pass --apply_zed_sim2real to the examples. The images then get the ZED Sim2Real camera model the streaming path applies, without motion blur. Each environment gets its own auto exposure and white balance.

In your own script, request rgb_hdr instead of rgb in the cameras’ data_types, and use sl.sensor.camera.zed_sim2real.ZedSim2Real on it:

from sl.sensor.camera.zed_sim2real import ZedSim2Real
sim2real = ZedSim2Real(num_envs=1, camera_model=args.camera_model, resolution=args.resolution)
sim2real.prepare_cameras(sim.stage, [paths["left"], paths["right"]])   # once, before sim.reset()
wait_for_camera_data(sim, [left, right], keys=("rgb_hdr", "distance_to_image_plane"))
# ... each step:
rgb_left, rgb_right = sim2real.apply(unwrap_output(left.data.output, "rgb_hdr"),
                                     unwrap_output(right.data.output, "rgb_hdr"), dt)

apply() returns (N, H, W, 3) uint8 images on the GPU.

Extension 5.3.0 changed this API: ZedSim2Real takes HDR images and returns new ones instead of modifying 8-bit tensors in place, and --zed_sim2real_scene_lux is gone.

Troubleshooting

  • Empty or all-zero frames (own scripts). --enable_cameras is missing: RGB and depth are render-based. The shipped examples force the flag on.
  • Wrong field of view (own scripts). Call apply_zed_optics() after placing the ZED USD, before the Camera sensors start.
  • data.pos_w stays at the spawn pose on a robot-mounted ZED. The images and depth follow the link; the reported pose is a known Isaac™ Lab quirk. Derive the live camera pose from the link’s body pose and the mount offset.
  • An example exits with unrecognized arguments. Since 5.3.0 the examples only accept the options they use: --apply_zed_sim2real is for zed_single.py and zed_multi_env.py, and zed_record_demos.py no longer takes --save_dir or --num_frames.
  • zed_record_demos.py stops with already exists. It refuses to overwrite an earlier recording: pass another --dataset_file, or move the file away.

For the full argument list, output formats, and the demo ports, see the examples README on GitHub.

See also