GStreamer - ZED Metadata

The stream of data injected in a GStreamer pipeline by the zedsrc element contains color and depth information plus metadata with sensors and object detection data. The zedxonesrc element attaches the same metadata structure to its buffers, with the object detection and sensors sections left empty.

The ZED metadata are defined in the GStreamer library gstzedmeta.

Metadata information

GstZedSrcMeta: metadata global container
ZedInfo: general information about the ZED camera that acquired the data.
ZedPose: Positional Tracking information.
ZedSensors: data from the camera sensors (if available).
ZedImu: 6 DOF inertial data (if available).
ZedMag: 3 DOF magnetic field data (if available).
ZedEnv: environmental data, atmospheric pressure and internal temperature (if available).
ZedCamTemp: camera CMOS temperatures (if available).
ZedObjectData: object detection data (if available).

Data Structures

GstZedSrcMeta

C++

struct _GstZedSrcMeta {
    GstMeta meta;
    ZedInfo info;
    ZedPose pose;
    ZedSensors sens;
    gboolean od_enabled;
    guint8 obj_count;
    guint64 frame_id;
    ZedObjectData objects[256];
};

meta: metadata identifier field.
info: camera information.
pose: camera pose.
sens: sensors data.
od_enabled: indicates if object detection data are available.
obj_count: the number of detected objects [max 256].
frame_id: track the meta/buffer throughout the GStreamer pipeline (when working with source code)
objects: array of detected objects.

ZedInfo

C++

struct _ZedInfo {
    gint cam_model;
    gint stream_type;
    guint grab_single_frame_width;
    guint grab_single_frame_height;
};

cam_model: camera model, i.e. the value of the sl::MODEL enum of the ZED SDK (0 : ZED, 1 : ZED Mini, 2 : ZED 2, 3 : ZED 2i, 4 : ZED X, 5 : ZED X Mini, 9 : ZED X Nano, 11 : Virtual ZED X, 30 : ZED X One GS, 31 : ZED X One 4K, 33 : ZED X One Core [ZED SDK v5.5 and newer]). The field is a direct copy of the sl::MODEL enum, so the set of possible values follows the ZED SDK release used to build the plugin.
stream_type: type of stream, i.e. the resolved value of the stream-type property of the source element (0 : Left image, 1 : Right image, 2 : Stereo couple up/down, 3 : 16-bit depth map, 4 : Left + Depth up/down, 5 : Stereo couple left/right). On NVIDIA® Jetson™ with ZED SDK v5.2 or newer the zero-copy NV12 values 6 (left), 7 (stereo side-by-side) and 8 (right) can also be reported.
grab_single_frame_width: original width of image data.
grab_single_frame_height: original height of image data.

ZedPose

C++

struct _ZedPose {
    gboolean pose_avail;
    gint pos_tracking_state;
    gfloat pos[3];
    gfloat orient[3];
};

pose_avail: indicates if camera pose is available.
pos_tracking_state: status of the Positional Tracking algorithm, i.e. the value of the sl::POSITIONAL_TRACKING_STATE enum of the ZED SDK (0 : SEARCHING [deprecated, no longer returned], 1 : OK, 2 : OFF, 3 : FPS_TOO_LOW, 4 : SEARCHING_FLOOR_PLANE, 5 : UNAVAILABLE)
pos: camera position
orient: camera orientation (Euler angles).

ZedSensors

C++

struct _ZedSensors {
    gboolean sens_avail;
    ZedImu imu;
    ZedMag mag;
    ZedEnv env;
    ZedCamTemp temp;
};

sens_avail: indicates if sensors data are available (all camera models with IMU support, i.e. every model except the original ZED).
imu: IMU data.
mag: Magnetometer data.
env: environment data.
temp: camera temperatures data

An example about how to retrieve Sensors data is provided with the gstzeddatacsvsink element source code

ZedImu

C++

struct _ZedImu {
    gboolean imu_avail;
    gfloat acc[3];
    gfloat gyro[3];
    gfloat temp;
};

imu_avail: indicates if IMU data are available (all camera models with IMU support, i.e. every model except the original ZED).
acc: 3 DOF accelerometer data in [m/s²].
gyro: 3 DOF gyroscope data in [rad/sec].
temp: IMU temperature in [°C].

ZedMag

C++

struct _ZedMag {
    gboolean mag_avail;
    gfloat mag[3];
};

mag_avail: indicates if magnetometer data are available (ZED 2 and ZED 2i only).
mag: 3 DOF magnetic field data in [µT].

ZedEnv

C++

struct _ZedEnv {
    gboolean env_avail;
    gfloat press;
    gfloat temp;
};

env_avail: indicates if environment data are available (ZED 2 and ZED 2i only).
press: atmospheric pressure in [hPa].
temp: internal camera temperature in [°C].

ZedCamTemp

C++

struct _ZedCamTemp {
    gboolean temp_avail;
    gfloat temp_cam_left;
    gfloat temp_cam_right;
};

temp_avail: indicates if CMOS temperatures data are available (ZED 2 and ZED 2i only).
temp_cam_left: temperature of the left CMOS sensor in [°C].
temp_cam_right: temperature of the right CMOS sensor in [°C].

ZedObjectData

C++

struct _ZedObjectData {
    gint id;
    OBJECT_CLASS label;
    OBJECT_SUBCLASS sublabel;
    OBJECT_TRACKING_STATE tracking_state;
    OBJECT_ACTION_STATE action_state;
    gfloat confidence;
    gfloat position[3];
    gfloat position_covariance[6];
    gfloat velocity[3];
    unsigned int bounding_box_2d[4][2];
    /* 3D bounding box of the person represented as eight 3D points
          1 ------ 2
         /        /|
        0 ------ 3 |
        | Object | 6
        |        |/
        4 ------ 7
    */
    gfloat bounding_box_3d[8][3];
    gfloat dimensions[3]; // 3D object dimensions: width, height, length
    gboolean skeletons_avail;
    gint skel_format; // indicates if it's 18, 34, 38, or 70 skeleton model
    gfloat keypoint_2d[70][2]; // Negative coordinates -> point not valid
    gfloat keypoint_3d[70][3]; // Nan coordinates -> point not valid
    gfloat head_bounding_box_2d[4][2];
    gfloat head_bounding_box_3d[8][3];
    gfloat head_position[3];
};

id: unique identifier of the tracked object.
label: class of the identified object.
sublabel: subclass of the identified object [only for MULTICLASS models]
tracking_state: tracking status of the object.
action_state: action status of the object (static or moving).
confidence: confidence level of the detection [0, 100].
position: 3D position of the center of the object.
position_covariance: covariance matrix of the position.
velocity: velocity of the object
bounding_box_2d: 2D image coordinates of the four corners of the bounding box
bounding_box_3d: 3D world coordinates of the eight corners of the bounding box
dimensions: 3D dimensions of the 3D bounding box
skeletons_avail: indicates if a skeleton tracking detection was enabled and if skeleton data are available
skel_format: number of keypoints of the body format in use (18, 34, 38 or 70), driven by the bt-format property of the zedsrc element, which currently exposes the 18, 34 and 38 keypoint formats. It tells how many entries of keypoint_2d/keypoint_3d are meaningful.
keypoint_2d: 2D image coordinates of the skeleton joints. The array is always dimensioned for the largest body format (70 joints); only the first skel_format entries are valid.
keypoint_3d: 3D world coordinates of the skeleton joints, with the same convention as keypoint_2d.
head_bounding_box_2d: 2D image coordinates of the four corners of the bounding box of the head
head_bounding_box_3d: 3D world coordinates of the eight corners of the bounding box of the head
head_position: 3D world coordinates of the position of the center of the head

C++

enum class OBJECT_CLASS {
    PERSON = 0,          /**< For people detection */
    VEHICLE = 1,         /**< For vehicle detection. It can be cars, trucks, buses, motorcycles etc */
    BAG = 2,             /**< For bag detection (backpack, handbag, suitcase) */
    ANIMAL = 3,          /**< For animal detection (cow, sheep, horse, dog, cat, bird, etc) */
    ELECTRONICS = 4,     /**< For electronic device detection (cellphone, laptop, etc) */
    FRUIT_VEGETABLE = 5, /**< For fruit and vegetable detection (banana, apple, orange, carrot, etc) */
    SPORT = 6,           /**< For sport-related object detection (sportball) */
    LAST
};
enum class OBJECT_SUBCLASS {
    PERSON = 0,       /**< PERSON / PERSON_BODY */
    PERSON_HEAD = 22, /**< PERSON */
    BICYCLE = 1,      /**< VEHICLE */
    CAR = 2,          /**< VEHICLE */
    MOTORBIKE = 3,    /**< VEHICLE */
    BUS = 4,          /**< VEHICLE */
    TRUCK = 5,        /**< VEHICLE */
    BOAT = 6,         /**< VEHICLE */
    BACKPACK = 7,     /**< BAG */
    HANDBAG = 8,      /**< BAG */
    SUITCASE = 9,     /**< BAG */
    BIRD = 10,        /**< ANIMAL */
    CAT = 11,         /**< ANIMAL */
    DOG = 12,         /**< ANIMAL */
    HORSE = 13,       /**< ANIMAL */
    SHEEP = 14,       /**< ANIMAL */
    COW = 15,         /**< ANIMAL */
    CELLPHONE = 16,   /**< ELECTRONICS */
    LAPTOP = 17,      /**< ELECTRONICS */
    BANANA = 18,      /**< FRUIT/VEGETABLE */
    APPLE = 19,       /**< FRUIT/VEGETABLE */
    ORANGE = 20,      /**< FRUIT/VEGETABLE */
    CARROT = 21,      /**< FRUIT/VEGETABLE */
    SPORTSBALL = 23,  /**< SPORT */
    LAST
};
enum class OBJECT_TRACKING_STATE {
    OFF,       /**< The tracking is not yet initialized, the object ID is not usable */
    OK,        /**< The object is tracked */
    SEARCHING, /**< The object couldn't be detected in the image and is potentially occluded, the trajectory is estimated */
    TERMINATE, /**< This is the last searching state of the track, the track will be deleted in the next retreiveObject */
    LAST
};
enum class OBJECT_ACTION_STATE {
    IDLE = 0,   /**< The object is staying static. */
    MOVING = 1, /**< The object is moving. */
    LAST
};

Skeleton body formats

The skeleton namespace defines one enumeration of body parts, one getIdx_* helper and one list of bones for each of the four body formats supported by the Body Tracking module. The format in use is reported by the skel_format field of ZedObjectData and is selected with the bt-format property of the zedsrc element.

Body format	Parts enum	Index helper	Bones list
18 keypoints	`BODY_18_PARTS`	`getIdx_18()`	`BODY_18_BONES`
34 keypoints	`BODY_34_PARTS`	`getIdx_34()`	`BODY_34_BONES`
38 keypoints	`BODY_38_PARTS`	`getIdx_38()`	`BODY_38_BONES`
70 keypoints	`BODY_70_PARTS`	`getIdx_70()`	`BODY_70_BONES`

The 18-keypoint format, for example, is defined as follows:

C++

namespace skeleton {
enum class BODY_18_PARTS {
    NOSE = 0,
    NECK = 1,
    RIGHT_SHOULDER = 2,
    RIGHT_ELBOW = 3,
    RIGHT_WRIST = 4,
    LEFT_SHOULDER = 5,
    LEFT_ELBOW = 6,
    LEFT_WRIST = 7,
    RIGHT_HIP = 8,
    RIGHT_KNEE = 9,
    RIGHT_ANKLE = 10,
    LEFT_HIP = 11,
    LEFT_KNEE = 12,
    LEFT_ANKLE = 13,
    RIGHT_EYE = 14,
    LEFT_EYE = 15,
    RIGHT_EAR = 16,
    LEFT_EAR = 17,
    LAST = 18
};
inline int getIdx_18(BODY_18_PARTS part) { return static_cast<int>(part); }
const std::vector<std::pair<BODY_18_PARTS, BODY_18_PARTS>> BODY_18_BONES{
    {BODY_18_PARTS::NOSE, BODY_18_PARTS::NECK},
    {BODY_18_PARTS::NECK, BODY_18_PARTS::RIGHT_SHOULDER},
    {BODY_18_PARTS::RIGHT_SHOULDER, BODY_18_PARTS::RIGHT_ELBOW},
    {BODY_18_PARTS::RIGHT_ELBOW, BODY_18_PARTS::RIGHT_WRIST},
    {BODY_18_PARTS::NECK, BODY_18_PARTS::LEFT_SHOULDER},
    {BODY_18_PARTS::LEFT_SHOULDER, BODY_18_PARTS::LEFT_ELBOW},
    {BODY_18_PARTS::LEFT_ELBOW, BODY_18_PARTS::LEFT_WRIST},
    {BODY_18_PARTS::RIGHT_SHOULDER, BODY_18_PARTS::RIGHT_HIP},
    {BODY_18_PARTS::RIGHT_HIP, BODY_18_PARTS::RIGHT_KNEE},
    {BODY_18_PARTS::RIGHT_KNEE, BODY_18_PARTS::RIGHT_ANKLE},
    {BODY_18_PARTS::LEFT_SHOULDER, BODY_18_PARTS::LEFT_HIP},
    {BODY_18_PARTS::LEFT_HIP, BODY_18_PARTS::LEFT_KNEE},
    {BODY_18_PARTS::LEFT_KNEE, BODY_18_PARTS::LEFT_ANKLE},
    {BODY_18_PARTS::RIGHT_SHOULDER, BODY_18_PARTS::LEFT_SHOULDER},
    {BODY_18_PARTS::RIGHT_HIP, BODY_18_PARTS::LEFT_HIP},
    {BODY_18_PARTS::NOSE, BODY_18_PARTS::RIGHT_EYE},
    {BODY_18_PARTS::RIGHT_EYE, BODY_18_PARTS::RIGHT_EAR},
    {BODY_18_PARTS::NOSE, BODY_18_PARTS::LEFT_EYE},
    {BODY_18_PARTS::LEFT_EYE, BODY_18_PARTS::LEFT_EAR}
};
}

The complete definition of the four formats is available in the gstzedmeta.h header.

An example about how to use Object Detection data is provided with the gstzedodoverlay element source code