Using the Body Tracking API

Since V4.0, the AI detection module has been split into two different modules: body tracking and object detection. Each module has its own data structures, methods and parameters. Before that, the body tracking feature was directly integrated into the object detection module.

Body Tracking Configuration

To configure the body tracking module, use BodyTrackingParameters at initialization and BodyTrackingRuntimeParameters to change specific parameters during use.

• new detection model BodyTrackingParameters::detection_model that enables human body detection. This preset configures the runtime and accuracy of the human body detector :

  • BODY_TRACKING_MODEL::HUMAN_BODY_FAST: real time performance even on NVIDIA® Jetson™ or low-end GPU cards

  • BODY_TRACKING_MODEL::HUMAN_BODY_MEDIUM: this is a compromise between accuracy and speed

  • BODY_TRACKING_MODEL::HUMAN_BODY_ACCURATE: state-of-the-art accuracy, requires powerful GPU

• BodyTrackingParameters::enable_body_fitting: this enables the fitting process of each detected person. This must be enabled to retrieve the local rotations of each keypoints. Otherwise, the data will be empty.

• BodyTrackingParameters::body_format is the body format outputted by the ZED SDK. The currently supported body formats are:

  • BODY_FORMAT::BODY_18 : 18 keypoints body model. This is a COCO18 format and is not directly compatible with public software like Unreal or Unity. For this reason, all local keypoints’ rotation and translation are not available with this format.

  • BODY_FORMAT::BODY_34 : 34 keypoints body model. This model is compatible with public software and all available data for BODY_18 can be extracted with this format. The body_fitting option must be enabled to use this format.

  • BODY_FORMAT::BODY_38 : 38 keypoints body model. This includes simplified hands and feet keypoints.

The code below shows how to set these new attributes.

1
// Set initialization parameters
2
BodyTrackingParameters detection_parameters;
3
detection_parameters.detection_model = BODY_TRACKING_MODEL::HUMAN_BODY_ACCURATE; //specific to human skeleton detection
4
detection_parameters.enable_tracking = true; // Objects will keep the same ID between frames
5
detection_parameters.enable_body_fitting = true; // Fitting process is called, user have access to all available data for a person processed by SDK
6
detection_parameters.body_format = BODY_FORMAT::BODY_34; // selects the 34 keypoints body model for SDK outputs
7
8
// Set runtime parameters
9
BodyTrackingRuntimeParameters detection_parameters_rt;
10
detection_parameters_rt.detection_confidence_threshold = 40;

If you want to track persons’ motion within their environment, you will first need to activate the positional tracking module. Then, set detection_parameters.enable_tracking to true.

1
if (detection_parameters.enable_tracking) {
2
    // Set positional tracking parameters
3
    PositionalTrackingParameters positional_tracking_parameters;
4
    // Enable positional tracking
5
    zed.enablePositionalTracking(positional_tracking_parameters);
6
}

With these parameters configured, you can enable the Body Tracking module:

1
// Enable body tracking with initialization parameters
2
zed_error = zed.enableBodyTracking(detection_parameters);
3
if (zed_error != ERROR_CODE::SUCCESS) {
4
    cout << "enableBodyTracking: " << zed_error << "\nExit program.";
5
    zed.close();
6
    exit(-1);
7
}

Getting Human Body Data

To get the detected person in a scene, get a new image with grab(...) and extract the detected person with retrieveBodies(). This process is exactly the same as getting new objects with the Object detection module.

1
sl::Bodies bodies; // Structure containing all the detected bodies
2
// grab runtime parameters
3
RuntimeParameters runtime_parameters;
4
runtime_parameters.measure3D_reference_frame = sl::REFERENCE_FRAME::WORLD;
5
6
if (zed.grab(runtime_parameters) == ERROR_CODE::SUCCESS) {
7
  zed.retrieveBodies(bodies, detection_parameters_rt); // Retrieve the detected bodies
8
}

sl::Bodies class stores all data regarding the different persons present in the scene in its vector<sl::BodyData> body_list attribute. Each person’s data are stored as a sl::BodyData. sl::Bodies also contain the timestamp of the detection, which can help connect the bodies to the images.

All 2D data are related to the left image, while the 3D data are either in the CAMERA or WORLD referential depending on RuntimeParameters.measure3D_reference_frame (given to the grab() function). The 2D data are expressed in the initial camera resolution RESOLUTION. A scaling can be applied if the value is needed in another resolution.

For 3D data, the coordinate frame and units can be set by the user using COORDINATE_SYSTEM and UNIT respectively. These settings are accessible using InitParameters when opening the ZED camera with the open function.

Accessing 2D and 3D body keypoints

Once a sl::BodyData is retrieved from the body vector, you can access information such as its ID, position, velocity, label, and tracking_state but also its keypoint positions and rotations.

2D and 3D keypoint data of a detected person are accessible in a vector of pixel keypoint keypoint_2d and a vector of 3D position keypoint.

1
// collect all 2D keypoints
2
for (auto& kp_2d : body.keypoint_2d) {
3
  // user code using each kp_2d point
4
}
5
6
// collect all 3D keypoints
7
for (auto& kp_3d : obj.keypoint)
8
{
9
  // user code using each kp_3d point
10
}

See the keypoint index and name correspondence as well as the output skeleton format here.

Getting more results

When the fitting is enabled at the initial configuration stage, more results are available according to the chosen BODY_FORMAT. All local rotation and translation of each keypoint become available to the user with BODY_FORMAT::BODY_34 or BODY_FORMAT::BODY_38 format.

1
// collect local rotation for each keypoint
2
for (auto &kp : body.local_orientation_per_joint)
3
{
4
   // kp is the local keypoint rotation represented by a quaternion
5
   // user code
6
}
7
8
// collect local translation for each keypoint
9
for (auto &kp : body.local_position_per_joint)
10
{
11
12
   // kp is the local keypoint translation
13
   // user code
14
}
15
16
// get global root orientation
17
auto global_root_orientation = body.global_root_orientation
18
19
// note that global root translation is available in body.keypoint[root_index] where root_index is the root index of the body model

Code Example

For code examples, check out the Tutorial and Sample on GitHub.