Using the Depth Sensing API

Depth Sensing Configuration

To enable depth sensing, set options in InitParameters when initializing the camera. For runtime adjustments—such as toggling depth computation or changing sensing modes—use RuntimeParameters while the camera is running.

1
// Set configuration parameters
2
InitParameters init_params;
3
init_params.depth_mode = DEPTH_MODE::ULTRA; // Use ULTRA depth mode
4
init_params.coordinate_units = UNIT::MILLIMETER; // Use millimeter units (for depth measurements)

For more information on depth configuration parameters, see Depth Settings.

Retrieving Depth Data

To obtain the depth map of a scene, first call grab() to capture a new frame, then use retrieveMeasure() to access the depth data aligned with the left image. The retrieveMeasure() function allows you to retrieve various types of data, including the depth map, confidence map, normal map, or point cloud, depending on the specified measure type.

1
sl::Mat image;
2
sl::Mat depth_map;
3
if (zed.grab() == ERROR_CODE::SUCCESS) {
4
  // A new image and depth is available if grab() returns SUCCESS
5
  zed.retrieveImage(image, VIEW::LEFT); // Retrieve left image
6
  zed.retrieveMeasure(depth_map, MEASURE::DEPTH); // Retrieve depth
7
}

Accessing Depth Values

The depth map is stored in a sl::Mat object, which acts as a 2D matrix where each element represents the distance from the camera to a specific point in the scene. Each pixel at coordinates (X, Y) contains a 32-bit floating-point value indicating the depth (Z) at that location, typically in millimeters unless otherwise configured.

To access the depth value at a particular pixel, use the getValue() method provided by the SDK. This allows you to retrieve the distance from the camera to the object at the specified pixel coordinates.

1
float depth_value=0;
2
depth_map.getValue(x, y, &depth_value);

By default, depth values are expressed in millimeters. Units can be changed using InitParameters::coordinate_units. Advanced users can retrieve images, depth and points clouds either in CPU memory (default) or in GPU memory using retrieveMeasure(*, *, MEM_GPU).

Displaying Depth Image

The 32-bit depth map can be displayed as a grayscale 8-bit image

To display the depth map, the ZED SDK scales the real depth values to 8-bit values [0, 255], where 255 (white) represents the closest possible depth value and 0 (black) represents the most distant possible depth value. We call this process depth normalization.

To retrieve a depth image, you can use retrieveImage(depth, VIEW::DEPTH).

1
sl::Mat depth_for_display;
2
zed.retrieveImage(depth_for_display, VIEW::DEPTH);

Getting Point Cloud Data

The ZED camera can also provide a 3D point cloud, which is a collection of points in 3D space representing the scene. Each point in the point cloud corresponds to a pixel in the depth map and contains its (X, Y, Z) coordinates along with color information (RGBA).

A 3D point cloud with (X,Y,Z) coordinates and RGBA color can be retrieved using retrieveMeasure().

1
sl::Mat point_cloud;
2
zed.retrieveMeasure(point_cloud, MEASURE::XYZRGBA);

To access a specific pixel value, use getValue().

1
float4 point3D;
2
// Get the 3D point cloud values for pixel (i, j)
3
point_cloud.getValue(i, j, &point3D);
4
float x = point3D.x;
5
float y = point3D.y;
6
float z = point3D.z;
7
float color = point3D.w;

The point cloud stores its data on 4 channels using a 32-bit float for each channel. The last float is used to store color information, where R, G, B, and alpha channels (4 x 8-bit) are concatenated into a single 32-bit float.

You can choose between different color formats using XYZ<COLOR>. For example, BGRA color is available using retrieveMeasure(point_cloud, MEASURE::XYZBGRA).

Measuring distance in point cloud

When measuring distances, use the 3D point cloud instead of the depth map. The Euclidean distance formula allows us to calculate the distance of an object relative to the left eye of the camera.

1
float4 point3D;
2
// Measure the distance of a point in the scene represented by pixel (i,j)
3
point_cloud.getValue(i, j, &point3D);
4
float distance = sqrt(point3D.x * point3D.x + point3D.y * point3D.y + point3D.z * point3D.z);

Getting Normal Map

Retrieving Surface Normals

You can obtain a normal map by calling retrieveMeasure() with the NORMALS measure type. Surface normals are useful for applications such as traversability analysis and real-time lighting, as they describe the orientation of surfaces in the scene.

The normal map is stored as a 4-channel, 32-bit floating-point matrix, where the X, Y, and Z components represent the direction of the normal vector at each pixel. The fourth channel is unused.

1
sl::Mat normal_map;
2
zed.retrieveMeasure(normal_map, MEASURE::NORMALS);

To access the normal vector at a specific pixel, use the getValue() method, which returns the (X, Y, Z) components of the normal.

Adjusting Depth Resolution

To optimize performance and reduce data acquisition time, you can retrieve depth or point cloud data at a lower resolution by specifying the desired width and height in the retrieveMeasure() function. Additionally, you can choose whether the data is stored in CPU (RAM) or GPU memory by setting the appropriate memory type parameter. This flexibility allows you to balance processing speed and resource usage according to your application’s needs.

1
sl::Mat point_cloud;
2
// Retrieve a resized point cloud
3
// width and height specify the total number of columns and rows for the point cloud dataset
4
width = zed.getResolution().width / 2;
5
height = zed.getResolution().height / 2;
6
zed.retrieveMeasure(point_cloud, MEASURE::XYZRGBA, MEM::GPU, width, height);

Code Example

For code examples, check out the Tutorial and Sample on GitHub.