> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.stereolabs.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.stereolabs.com/_mcp/server.

# Using the ZED in Isaac™ Lab

[Isaac™ Lab](https://isaac-sim.github.io/IsaacLab/) is NVIDIA®'s framework for robot learning built on top of Isaac™ Sim. The ZED Camera extension lets you use a simulated ZED camera directly inside Isaac™ Lab and read its **stereo RGB pair** and a **depth** map as in-process tensors, with **no ZED SDK and no network streaming**.

This is the path for **reinforcement learning** and **imitation learning** workflows, where you want camera observations as tensors on the GPU rather than a ZED SDK stream. For how it compares to the streaming and SDK-free depth paths, see [Choosing a data path](/docs/integrations/isaac-sim#choosing-a-data-path).

> **Note**
>
> The depth here is Isaac™ Sim's **simulated renderer depth** (`distance_to_image_plane`), not the ZED SDK's stereo-matched depth. For genuine ZED SDK depth, use the streaming path.

## How it works

The ZED USD model (e.g. `ZED_X.usdc`) has authored `CameraLeft` / `CameraRight` prims with the correct intrinsics and stereo baseline. In Isaac™ Lab you reference that USD, attach an `isaaclab.sensors.Camera` sensor to each of those prims, and read `rgb` and `distance_to_image_plane` each simulation step. To place or mount the camera, you move the **whole USD root** so the stereo rig moves rigidly with it; you never transform the individual camera prims.

## Requirements

* An [Isaac™ Lab](https://isaac-sim.github.io/IsaacLab/) installation (verified on Isaac™ Lab 3.0 / Kit 110.1.2).
* The **ZED Camera extension** (from the zed-isaac-sim repository) on the Python path. The shipped examples handle this automatically via a small bootstrap import.
* Launch through the Isaac™ Lab launcher (`isaaclab.bat` / `isaaclab.sh`). The shipped examples enable cameras automatically; when launching your own scripts, pass **`--enable_cameras`**: RGB and depth are render-based, so without it the renderer is off and frames are empty.

> **Warning**
>
> The **first** rendered frame compiles RTX shaders and can take several minutes on a cold cache. It is compiling, not hung; subsequent runs are fast.

## Quick start

The extension ships ready-to-run examples under [`exts/sl.sensor.camera/examples/isaaclab/`](https://github.com/stereolabs/zed-isaac-sim/tree/main/exts/sl.sensor.camera/examples/isaaclab). To add a single ZED X to a scene and read its stereo RGB + depth tensors:

```bash
# Windows
isaaclab.bat -p exts/sl.sensor.camera/examples/isaaclab/zed_single.py --resolution SVGA --save_dir C:\zed_lab_out

# Linux
./isaaclab.sh -p exts/sl.sensor.camera/examples/isaaclab/zed_single.py --resolution SVGA --save_dir /tmp/zed_lab_out
```

Other examples include a vectorized multi-environment capture (`zed_multi_env.py`) and ports of flagship Isaac™ Lab demos (walking quadruped, Franka lift, imitation-learning HDF5 recording) with the actual ZED\_X model mounted on the robot.

*Franka lift task with ZED observations.*

## Using the sensor in your own script

The `sl.sensor.camera` package is importable without enabling the Kit extension (a plain `sys.path` insert is enough), and its helpers do the ZED-specific work:

```python
import isaaclab.sim as sim_utils
from isaaclab.sensors import Camera
from sl.sensor.camera.isaaclab_utils import make_camera_cfg, unwrap_output, wait_for_camera_data
from sl.sensor.camera.utils import get_camera_paths, get_camera_usd_path

# 1) Reference the ZED USD and place the WHOLE root (the cameras inherit the pose + baseline).
root = "/World/ZED_X"
zed = sim_utils.UsdFileCfg(usd_path=get_camera_usd_path("ZED_X"))
zed.func(root, zed)

# 2) Attach Camera sensors to the authored stereo prims.
paths = get_camera_paths(root, "ZED_X")   # {"left": .../CameraLeft, "right": ..., "imu": ...}
left  = Camera(make_camera_cfg(paths["left"],  "ZED_X", "SVGA",
                               data_types=["rgb", "distance_to_image_plane"]))
right = Camera(make_camera_cfg(paths["right"], "ZED_X", "SVGA", data_types=["rgb"]))

# 3) Each step, read the tensors.
rgb_left = unwrap_output(left.data.output, "rgb")                       # (num_cameras, H, W, 3)
depth    = unwrap_output(left.data.output, "distance_to_image_plane")   # (num_cameras, H, W, 1), meters
```

> **Note**
>
> When mounting the ZED on a robot, transform the **USD root** (or parent it under a robot link); never move the `CameraLeft` / `CameraRight` prims individually, or you break the stereo rig.

## ZED Sim2Real camera model (experimental)

> **Note**
>
> This feature is experimental and under active development; its behavior and calibration may change.

To make the RGB tensors look like a real ZED X camera (lens blur, vignetting, auto-exposure, color, sensor noise) instead of a clean render, pass **`--apply_zed_sim2real`** to the examples (optionally with `--zed_sim2real_scene_lux <lux>`). Each environment gets its own auto-exposure state, and the RGB tensors are degraded in place, which is useful for closing the sim-to-real gap in RL / imitation-learning pipelines. This is the same calibrated model the streaming path applies; it runs on the GPU via the `sl_zed_sim2real` library (a safe no-op if that library is not installed).

For the full argument list, output formats, and the flagship demo ports, see the [examples README on GitHub](https://github.com/stereolabs/zed-isaac-sim/tree/main/exts/sl.sensor.camera/examples/isaaclab).

## See also

* **[Setting up the ZED in Isaac™ Sim](/docs/integrations/isaac-sim/setting-up-the-zed-in-isaac-sim)**: stream the virtual camera to the ZED SDK instead of reading tensors in-process.
* **[SDK-free depth capture](/docs/integrations/isaac-sim/sdk-free-depth-capture)**: capture the same renderer depth to disk in plain Isaac™ Sim, with no Isaac™ Lab and no ZED SDK.