ProductsCamerasZED X One Core - MIPI

Setting up the ZED X One Core Stereo

Open in ClaudeOpen in ChatGPT

Two ZED X One Core cameras can be combined to create a Virtual Stereo Vision system, functioning as a single stereoscopic camera similar to the ZED 2i or ZED X.

This guide covers the MIPI CSI-2 ZED X One Core on a ZED Link MIPI capture card. If you are building a rig with GMSL2 ZED X One GS or ZED X One 4K cameras, follow Setting up the ZED X One Stereo instead.

Frame synchronization

When creating a stereo rig, it is essential that the two cameras provide synchronized image capture to ensure accurate depth perception.

The ZED Link MIPI Quad embeds an FPGA that synchronizes its four channels in real time, so a stereo pair is hardware-synchronized with a precision of 15 µs on any two of the four channels. Unlike a GMSL2 rig, no jumper, wiring, or synchronization cable is required: connect the two cameras to two channels of the same card and the frames are aligned.

Although frames are hardware-synchronized with 15 µs precision, software timestamps are assigned upon driver receipt and may exhibit slight millisecond-level jitter. Frames are considered synchronized if the timestamp difference is less than one frame period (1000/FPS ms).

On an NVIDIA® Jetson AGX Orin™ Developer Kit, where the four channels are available, a single ZED Link MIPI Quad can drive two independent stereo pairs, or one stereo pair plus two monocular cameras.

With the NVIDIA® Holoscan Sensor Bridge on a Jetson AGX Thor™, only the J1 and J2 channels are wired through. That configuration supports one stereo pair, and the two cameras must be connected to J1 and J2. See the Jetson AGX Thor setup guide.

Choose the baseline

The distance between the two cameras, the baseline, together with the lens option, determines the usable depth range: a small baseline sees accurately at very close range, while a larger baseline extends the range but increases the minimum working distance.

The following relations let you size the rig. For a fixed minimum depth value h, the required baseline B is:

B=(2∗h∗tan(α/2))B = (2 * h * tan(α/2))

Conversely, if you fix the baseline B, the corresponding minimum depth value h is:

h=(B/(2∗tan(α/2)))h = (B / (2 * tan(α/2)))

Where α is the horizontal Field of View of the lens option fitted to your cameras:

Lens optionFocal lengthHorizontal FOV (α)
Fisheye lens1.38 mm204°
Wide lens2.2 mm110°
Narrow lens4.6 mm73°

To obtain a minimum depth value that allows for a usable depth map, it is required to apply a “6x” factor to h. This guarantees that the fields of view of the two cameras have a good overlapping area:

h′=6∗h;B′=B/6h' = 6*h ; B' = B/6

The maximum depth (in meters) can be calculated with the formula:

Dmax=(f∗B)/(s∗disparitymin)D_max = (f * B) / (s * disparity_min)

Where:

  • B is the baseline in meters
  • f is the focal length of the optics in meters
  • s is the pixel size of the CMOS sensor in meters
  • disparity_min is the smallest disparity the matcher can resolve, in pixels

The complete optical and sensor figures are listed on the ZED X One Core Specifications page.

Build the stereo rig

The two cameras must be rigidly mounted parallel to each other at a fixed distance. It is critical that they do not move in rotation or translation relative to each other over time: any small movement requires a new calibration. The mechanical stability must be adapted to the usage conditions, especially vibrations or shocks that could affect alignment and significantly decrease depth accuracy.

The ZED X One Core is a barebone module intended for OEM integration, so the rig is part of your own mechanical design. Each camera provides 4x Ø2.2 mm mounting holes on a 27 x 27 mm body; refer to the technical drawings on the Specifications page for the mounting pattern and the position of the optical center of each lens option.

Cameras purchased with optics are factory focused, so no focusing step is needed. If you use cameras purchased without optics, focus both of them at the working distance of your application and lock the optics in place before starting the calibration procedure. Refocusing a lens afterwards invalidates the calibration.

The ZED X One Core does not support hot-plugging. Cameras must be connected and disconnected only when the host is powered off, and any change to the hardware configuration requires a full reboot of the host to take effect.

Install the software

You need two main software components to operate a dual ZED X One Core stereo system:

  1. The ZED Link MIPI driver, to configure the capture pipeline and retrieve images and inertial data.
  2. The ZED SDK, to process the stereoscopic data.

The ZED Link MIPI driver is distributed separately from the GMSL2 ZED Link driver, and the two are not interchangeable. Download the package matching your capture card and your Jetson™ Linux (L4T) version from the ZED X Camera Drivers page, install it, and reboot the Developer Kit.

The full assembly and driver installation procedure is described in the ZED Link MIPI Quad on AGX Orin setup guide. For troubleshooting, refer to the MIPI Troubleshooting guide.

Install the ZED SDK

The ZED SDK enables processing of stereoscopic data from two ZED X One Core cameras configured as a virtual stereo system. The sl::Camera class (C++ | Python | C# | C) provides the core functionality for depth computation and 3D perception.

To get started, download and install the ZED SDK on your NVIDIA® Jetson™ platform.

Verify that both cameras are enumerated:

ZED_Explorer --all

Calibrate the stereo rig

The calibration step is mandatory to perform depth estimation using the ZED SDK API. Each camera carries its own factory intrinsic calibration, but the geometry of the rig is specific to your assembly, so there is no factory stereo calibration for it: the procedure must be run once the two cameras are mounted.

The procedure determines:

  • Extrinsic parameters: the relative position and orientation between the two cameras in the rig, which is what makes depth computation possible.
  • Intrinsic parameters: the optical characteristics of each camera (focal length, optical center, distortion), recomputed by the same tool.

The calibration is performed with the open source zed-opencv-calibration tools, using the same procedure as for a ZED X One rig. Follow the stereo calibration procedure, which covers the checkerboard requirements, the acquisition workflow, and the validation step.

When calibrating a virtual stereo rig, the serial number of the Virtual Stereo Camera is generated by the ZED SDK from the serial numbers of the two individual cameras. Make sure to use this generated serial number when loading the calibration in your application, so the virtual stereo setup has a unique identifier.

Use the stereo rig with the ZED SDK

The ZED SDK includes built-in support for data acquisition from dual ZED X One stereo systems, allowing you to open and manage the virtual stereo rig directly through the SDK.

Verify first that both cameras are properly connected and recognized, with ZED_Explorer --all or ZED_Studio --list.

To open the rig, specify the serial numbers or the camera IDs of both cameras in the InitParameters when initializing the sl::Camera object:

#include <sl/Camera.hpp>
using namespace sl;
InitParameters init_params;
// Generate a unique virtual stereo serial number from the two ZED X One Core serial numbers
unsigned int sn_left = 123456789;  // Serial number of the left camera
unsigned int sn_right = 987654321; // Serial number of the right camera
int sn_stereo = sl::generateVirtualStereoSerialNumber(sn_left, sn_right);
init_params.input.setVirtualStereoFromSerialNumbers(sn_left, sn_right, sn_stereo);
// Set any other InitParameters as needed to configure the camera and depth settings
sl::Camera zed;
ERROR_CODE err = zed.open(init_params);
if (err != ERROR_CODE::SUCCESS)
    exit(-1);
Mat image, depth;
while (!exit_app) {
    if (zed.grab() == ERROR_CODE::SUCCESS) {
        zed.retrieveImage(image, VIEW::LEFT);
        zed.retrieveMeasure(depth, MEASURE::DEPTH);
        // Any processing
    }
}
// Close the camera
zed.close();

The full API surface, including the camera-ID variant of the initialization, is documented on the ZED X One Stereo page and applies unchanged to the ZED X One Core.

Example applications

The ZED SDK GitHub repository includes several example applications demonstrating how to use a dual ZED X One stereo system.

Use the stereo rig with ROS 2

The ZED ROS 2 wrapper supports dual ZED X One stereo systems.

When launching the ZED ROS 2 node, set virtual as the camera model and specify the camera IDs or the serial numbers of both cameras:

ros2 launch zed_wrapper zed_camera.launch.py camera_model:=virtual camera_ids:=[0,1]

or

ros2 launch zed_wrapper zed_camera.launch.py camera_model:=virtual camera_serial_numbers:=[123456789,987654321]

For more details, refer to the ZED ROS 2 wrapper documentation.

Develop on a PC

Once the rig is running on the Jetson™ device, you can stream the stereo data over the local network and develop against it from a desktop machine. See Developing with ZED X One Core on a PC.