Plane Detection Overview

Introduction

With the 3D information of the environment, the ZED camera can estimate the position of the planes present in the scene.

To detect a plane the position tracking needs to be enabled with zed.enablePositionalTracking(). Also, a plane can be detected only if the tracking state of the camera is OK.

Therefore the procedure for detecting a plane is as follows:

• Enable the positional tracking module
• Grab an image from the ZED
• Check that the tracking state is OK
• Estimate a plane position.

Detecting Planes

To estimate a plane position, you will need to use the findPlaneAtHit function and pass the 2D coordinates of a target pixel where you want to find its support plane. Below is an example of code that can be used to estimate the plane passing by the depth value of pixel coord = (u,v):

1
sl::Plane plane;
2
sl::uint2 coord; // Fill it with the coordinates taken from the full size image  
3
while (zed.grab() == ERROR_CODE::SUCCESS) {
4
  tracking_state = zed.getPosition(pose); // Get the tracking state of the camera
5
  if (tracking_state == TRACKING_STATE::OK) {  
6
    // Detect the plane passing by the depth value of pixel coord
7
    find_plane_status = zed.findPlaneAtHit(coord, plane);
8
  }
9
}

If it succeeds the function stores the detected plane in a sl::Plane object, which includes useful information such as 3D position, normal, polygon boundaries and plane type (vertical/horizontal).

Accessing Plane Data

The sl::Plane class contains all the information for defining the plane in space such as normal, center and equation. To access this information use the getter of the class. For example:

1
if (find_plane_status == ERROR_CODE::SUCCESS) {
2
  sl::float3 normal = plane.getNormal(); // Get the normal vector of the detected plane
3
  sl::float4 plane_equation = plane.getPlaneEquation(); // Get (a,b,c,d) where ax+by+cz+d=0
4
}

For AR purposes it can be useful to get the Transform of the detected plane according to the global reference frame. For example to place an object close to a wall or to transform the global reference frame center to the plane center. This transform is accessible as follows:

1
if (find_plane_status == ERROR_CODE::SUCCESS) {
2
  // Get the transform of the plane according to the global reference frame
3
  sl::Transform plane_transform = plane.getPose();
4
}

Convert Plane to Mesh

The detected plane can be converted to a Mesh. Converting it to a mesh can be useful when highlighting a plane in the scene. To get the mesh of the plane, call:

1
 sl::Mesh mesh = plane.extractMesh();

Detecting Floor Plane

With the 3D information of the environment, ZED cameras can estimate where is the ground floor in a scene.

Getting Floor Plane

The floor plane can be automatically detected by calling findFloorPlane instead of findPlaneAtHit. Apart from storing the plane in a sl::Plane, this function will also store the transform between the floor plane frame and the camera frame.

This transform can be used to reset the tracking and align it with the floor plane frame as follow:

1
sl::Plane plane;
2
sl::Transform resetTrackingFloorFrame;
3
find_plane_status = zed.findFloorPlane(plane, resetTrackingFloorFrame);
4
if (find_plane_status == ERROR_CODE::SUCCESS) {
5
  // Reset positional tracking to align it with the floor plane frame
6
  zed.resetPositionalTracking(resetTrackingFloorFrame);
7
}

This code can be simplified by using PositionalTrackingParameters::set_floor_as_origin to align the positional tracking reference frame on the ground floor.

1
PositionalTrackingParameters positional_tracking_parameters;
2
positional_tracking_parameters.set_floor_as_origin = true;
3
zed.enablePositionalTracking(positional_tracking_parameters);
4
5
RuntimeParameters runtime_parameters;
6
runtime_parameters.measure3D_reference_frame = REFERENCE_FRAME::WORLD;
7
sl::Mat cloud;
8
while (zed.grab(runtime_parameters) == ERROR_CODE::SUCCESS) {
9
  zed.retrieveMeasure(cloud, MEASURE::XYZRGBA);
10
  // The point cloud is aligned on the floor plane.
11
  // A threshold on the height could then be used as a simple object detection method
12
}

The estimation of the floor plane can be refined by passing prior parameters (such as prior height or orientation) to the findFloorPlane function. For additional details on these parameters, please check out the API Reference.