Getting Started

Introduction

In this tutorial, you will learn how to to capture and display color and depth images using OpenCV and the ZED SDK.

Sample Code

Sample code is available on GitHub and in the samples/interfaces directory of the SDK.

A pre-compiled version is available on Windows in the ZED SDK\samples\bin directory. Linux users can compile it with CMake, please follow the instructions.

Sharing image data between ZED SDK and OpenCV

The ZED SDK provides its own sl::Mat class to store image data, with a structure and data types similar to OpenCV cv::Mat. Image data can be shared between sl::Mat and cv::Mat by having the both memory pointers pointing to the same address. Threrefore, no data transfer is required between the two matrices.

To convert an sl::Mat to a cv::Mat, we provide an slMat2cvMat() function in our OpenCV sample.

// Create an RGBA sl::Mat object
sl::Mat image_zed(zed.getResolution(), MAT_TYPE_8U_C4);
// Create an OpenCV Mat that shares sl::Mat data
cv::Mat image_ocv = slMat2cvMat(image_zed);

Capturing Video

To capture video, use grab() and retrieveImage(). Since sl::Mat and cv::Mat share the same data, calling retrieveImage() will update the OpenCV matrix as well. Display the video with OpenCV using cv::imshow().

if (zed.grab() == SUCCESS) {
    // Retrieve the left image in sl::Mat
    // The cv::Mat is automatically updated
    zed.retrieveImage(image_zed, VIEW_LEFT);
    // Display the left image from the cv::Mat object
    cv::imshow("Image", image_ocv);
}

Capturing Depth

A depth map is a 1-channel matrix with 32-bit float values for each pixel. Each value expresses the distance of a pixel in the scene. The depth map can be retrieved using retrieveMeasure() and shared with a cv::Mat.

// Create a sl::Mat with float type (32-bit)
sl::Mat depth_zed(zed.getResolution(), MAT_TYPE_32F_C1);
// Create an OpenCV Mat that shares sl::Mat data
cv::Mat depth_ocv = slMat2cvMat(depth_zed);

if (zed.grab() == SUCCESS) {
    // Retrieve the depth measure (32-bit)
    zed.retrieveMeasure(depth_zed, MEASURE_DEPTH);
    // Print the depth value at the center of the image
    std::cout << depth_ocv.at<float>(depth_ocv.rows/2, depth_ocv.cols/2) << std::endl;
}

Displaying Depth

A cv::Mat with 32-bit float values can’t be displayed with cv::imshow. To display the depth map, we need to normalize the depth values between 0 and 255 (8-bit) and create a black and white representation. Do not use this representation for other purposes than displaying the image.

// Create an RGBA sl::Mat object
sl::Mat image_depth_zed(zed.getResolution(), MAT_TYPE_8U_C4);
// Create an OpenCV Mat that shares sl::Mat data
cv::Mat image_depth_ocv = slMat2cvMat(image_depth_zed);

if (zed.grab() == SUCCESS) {
    // Retrieve the normalized depth image
    zed.retrieveImage(image_depth_zed, VIEW_DEPTH);
    // Display the depth view from the cv::Mat object
    cv::imshow("Image", image_depth_ocv);  
}

UVC Capture

You can also use the ZED as a standard UVC camera in OpenCV to capture raw stereo video using the code snippet below. To get rectified images and calibration with OpenCV, use the native capture sample available on GitHub.

// Open the ZED camera
VideoCapture cap(0);
if(!cap.isOpened())
    return -1;

// Set the video resolution to HD720 (2560*720)
cap.set(CV_CAP_PROP_FRAME_WIDTH, 2560);
cap.set(CV_CAP_PROP_FRAME_HEIGHT, 720);

for(;;)
{
    Mat frame, left, right;
    // Get a new frame from camera
    cap >> frame;
    // Extract left and right images from side-by-side
    left_image = frame(cv::Rect(0, 0, frame.cols / 2, frame.rows));
    right_image = frame(cv::Rect(frame.cols / 2, 0, frame.cols / 2, frame.rows));
    // Display images    
    imshow("frame", frame);
    imshow("left", left_image);
    imshow("right", right_image);
    if(waitKey(30) >= 0) break;
}
// Deinitialize camera in the VideoCapture destructor
return 0;