How to Use OpenCV with ZED in C++

Introduction

In this tutorial, you will learn how to capture and display color and depth images using OpenCV and the ZED SDK in C++.

Code Example

The sample code is available on GitHub.

The ZED SDK provides its own sl::Mat class to store image data, with a structure and data types similar to OpenCV’s cv::Mat. Image data can be shared between sl::Mat and cv::Mat by having both memory pointers pointing to the same address. Therefore, no data transfer is required between the two matrices.

To convert an sl::Mat to a cv::Mat, we provide an slMat2cvMat() function in the OpenCV sample.

C++

// Create a sl::Mat object (4 channels of type unsigned char) to store the image.
sl::Mat image_zed(zed.getCameraInformation().camera_configuration.resolution, MAT_TYPE::U8_C4);
// Create an OpenCV Mat that shares sl::Mat data
cv::Mat image_ocv = slMat2cvMat(image_zed);

Capturing Video

To capture video, use grab() and retrieveImage(). Since sl::Mat and cv::Mat share the same data, calling retrieveImage() will update the OpenCV matrix as well. Display the video with OpenCV using cv::imshow().

C++

if (zed.grab() == ERROR_CODE::SUCCESS) {
    // Retrieve the left image in sl::Mat
    // The cv::Mat is automatically updated
    zed.retrieveImage(image_zed, VIEW::LEFT);
    // Display the left image from the cv::Mat object
    cv::imshow("Image", image_ocv);
}

Capturing Depth

A depth map is a 1-channel matrix with 32-bit float values for each pixel. Each value expresses the distance of a pixel in the scene. The depth map can be retrieved using retrieveMeasure() and shared with a cv::Mat. Please refer to the Depth API for more information.

C++

// Create a sl::Mat with float type (32-bit)
sl::Mat depth_zed(zed.getCameraInformation().camera_configuration.resolution, MAT_TYPE::F32_C1);
// Create an OpenCV Mat that shares sl::Mat data
cv::Mat depth_ocv = slMat2cvMat(depth_zed);
if (zed.grab() == ERROR_CODE::SUCCESS) {
    // Retrieve the depth measure (32-bit)
    zed.retrieveMeasure(depth_zed, MEASURE::DEPTH);
    // Print the depth value at the center of the image
    std::cout << depth_ocv.at<float>(depth_ocv.rows/2, depth_ocv.cols/2) << std::endl;
}

Displaying Depth

A cv::Mat with 32-bit float values can’t be displayed with cv::imshow. If you want to display the depth map you have to normalize it to fit in an unsigned char matrix. A call to sl::Camera::retrieveImage with VIEW::DEPTH returns such an image of the depth data as a grayscale image. Do not use this representation for other purposes than displaying the image.

C++

// Create a sl::Mat object (4 channels of type unsigned char) to store the image.
sl::Mat image_depth_zed(zed.getCameraInformation().camera_configuration.resolution, MAT_TYPE::U8_C4);
// Create an OpenCV Mat that shares sl::Mat data
cv::Mat image_depth_ocv = slMat2cvMat(image_depth_zed);
if (zed.grab() == ERROR_CODE::SUCCESS) {
    // Retrieve the normalized depth image
    zed.retrieveImage(image_depth_zed, VIEW::DEPTH);
    // Display the depth view from the cv::Mat object
    cv::imshow("Image", image_depth_ocv);  
}

UVC Capture

You can also use a USB ZED camera as a standard UVC camera in OpenCV to capture raw stereo video using the code snippet below (GMSL2 cameras such as the ZED X are not UVC devices). To get rectified images and calibration with OpenCV, use the native capture sample available on GitHub.

C++

// Open the ZED camera
cv::VideoCapture cap(0);
if(!cap.isOpened())
    return -1;
// Set the video resolution to HD720 (2560*720)
cap.set(cv::CAP_PROP_FRAME_WIDTH, 2560);
cap.set(cv::CAP_PROP_FRAME_HEIGHT, 720);
for(;;)
{
    cv::Mat frame, left_image, right_image;
    // Get a new frame from camera
    cap >> frame;
    // Extract left and right images from side-by-side
    left_image = frame(cv::Rect(0, 0, frame.cols / 2, frame.rows));
    right_image = frame(cv::Rect(frame.cols / 2, 0, frame.cols / 2, frame.rows));
    // Display images    
    cv::imshow("frame", frame);
    cv::imshow("left", left_image);
    cv::imshow("right", right_image);
    if(cv::waitKey(30) >= 0) break;
}
// Deinitialize camera in the VideoCapture destructor
return 0;