Software Integration

Getting Started with GStreamer and ZED

Open in ClaudeOpen in ChatGPT

GStreamer is a popular framework used to create custom media pipelines by combining modular plugins. The ZED GStreamer plugins greatly simplify the use of the ZED camera and SDK in a GStreamer media pipeline.

The ZED GStreamer package is composed of six plugins providing six elements, a metadata library, and a Linux RTSP Server application.

Plugins

NameClassificationSource CodeDescription
zedsrcSourceGitHubacquires the camera color image, depth map, sensor data, and object detection information and pushes them into a GStreamer pipeline.
zedxonesrc*SourceGitHubacquires ZED X One monocular camera color images and pushes them into a GStreamer pipeline.
zeddemuxStream DemuxerGitHubreceives a composite zedsrc stream (color left + color right + metadata or color left + depth map + metadata), processes the depth data when present, and pushes the result into two separate new streams named src_left and src_aux. A src_mono pad handles monocular zedxonesrc streams, and a src_data pad exposes the metadata for external processing.
zeddatamuxMetadata MuxerGitHubreceives a video stream compatible with ZED caps and a ZED Data Stream generated by zeddemux, then adds the metadata to the video stream. This is useful if the metadata is removed by a filter that does not automatically propagate it.
zeddatacsvsinkFile SinkGitHubsink element that receives ZED metadata, extracts the Positional Tracking and Sensors Data information, and saves it to a CSV file. Demonstrates how to retrieve and parse Positional Tracking and Sensors metadata.
zedodoverlayFilter Effect VideoGitHubtransform filter element that receives a ZED combined stream with metadata, extracts object detection information, and draws the overlays on the incoming image frames. Demonstrates how to retrieve and parse Object Detection metadata.

* Only built and installed on NVIDIA® Jetson™ devices with GMSL2 capabilities, running L4T 35.3/35.4 or L4T 36.3 and newer

zedodoverlay is built only if OpenCV is available on the build machine, and the RTSP Server application is built only if the GStreamer RTSP Server development package is available. The CMake configuration step prints the list of the optional components that were found.

Libraries

NameClassificationSource CodeDescription
zedmetaMetadata LibraryGitHubGStreamer library to define and handle the ZED metadata (Positional Tracking data, Sensors data, Detected Object data, Detected Skeletons data).

Applications

NameClassificationSource CodeDescription
RTSP ServerServer applicationGitHubLinux application that instantiates an RTSP server from a “gst-launch”-style text launch pipeline.

Installation

Prerequisites

Windows installation

  • Install the latest ZED SDK from the official download page [Optional to compile the zedsrc plugin to acquire data from a ZED camera device]

  • Install Git

  • Install CMake

  • Install a GStreamer distribution (both runtime and development installers).

  • The installer should set the installation path via the GSTREAMER_1_0_ROOT_X86_64 environment variable.

  • Add the path %GSTREAMER_1_0_ROOT_X86_64%\bin to the system variable PATH

  • [Optional] Install OpenCV to build the zedodoverlay filter

  • Run the following commands from a terminal or command prompt, assuming CMake and Git are in your PATH.

    git clone https://github.com/stereolabs/zed-gstreamer.git
    cd zed-gstreamer
    mkdir build
    cd build
    cmake -G "Visual Studio 16 2019" ..
    cmake --build . --target INSTALL --config Release

Linux installation

Install prerequisites

  • Install the latest ZED SDK from the official download page

  • Update list of apt available packages

    sudo apt update

  • Install GCC compiler and build tools

    sudo apt install build-essential

  • Install CMake build system

    sudo apt install cmake

  • Install GStreamer, the development packages and useful tools:

    sudo apt install \
    libgstreamer1.0-0 \
    gstreamer1.0-libav \
    libgstrtspserver-1.0-0 \
    gstreamer1.0-tools \
    gstreamer1.0-x \
    gstreamer1.0-alsa \
    gstreamer1.0-gl \
    gstreamer1.0-gtk3 \
    gstreamer1.0-qt5 \
    gstreamer1.0-pulseaudio \
    libgstreamer1.0-dev \
    libgstrtspserver-1.0-dev \
    libgstreamer-plugins-base1.0-0 \
    libgstreamer-plugins-base1.0-dev \
    libgstreamer-plugins-good1.0-0 \
    libgstreamer-plugins-good1.0-dev \
    libgstreamer-plugins-bad1.0-0 \
    libgstreamer-plugins-bad1.0-dev

  • [Optional] Install OpenCV to build the zedodoverlay filter

    sudo apt install libopencv-dev libopencv-contrib-dev

Clone the repository

git clone https://github.com/stereolabs/zed-gstreamer.git
cd zed-gstreamer

Build

mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make # Note: do not use the `-j` flag because parallel build is not supported
sudo make install

Installation test

Open a command line console and try the following commands to verify that everything is correctly installed:

  • Check ZED Video Source Plugin installation inspecting its properties:

    gst-inspect-1.0 zedsrc

  • Check ZED X One Video Source Plugin installation inspecting its properties (only Jetson™):

    gst-inspect-1.0 zedxonesrc

  • Check ZED Video Demuxer installation inspecting its properties:

    gst-inspect-1.0 zeddemux

  • Check ZED Data Mux Plugin installation inspecting its properties:

    gst-inspect-1.0 zeddatamux

  • Check ZED CSV Sink Plugin installation inspecting its properties:

    gst-inspect-1.0 zeddatacsvsink

  • Check ZED Object Detection Overlay Plugin installation inspecting its properties:

    gst-inspect-1.0 zedodoverlay

Ready-to-use scripts

The scripts folder of the repository contains ready-to-use pipelines for Windows, desktop Linux and Jetson:

  • simple-fps_rendering, simple-depth-fps_rendering: color and depth rendering with default settings
  • local-rgb_left_right-fps_rendering: left and right image rendering
  • local-rgb_left_depth-fps_rendering: left and depth image rendering
  • local-rgb-depth-sens-csv: left and depth image rendering, with sensors data saved to a CSV file
  • local-rgb-od_multi-overlay: left image rendering with MULTI_CLASS object detection overlay
  • local-od-fps_overlay: left and depth image rendering with FAST body tracking overlay
  • local-rgb-skel_fast-overlay, local-rgb-skel_accurate-overlay: body pose overlay with the FAST and ACCURATE models
  • local-rgb-rescale-od-overlay: left and depth image rendering with object detection and a rescaling filter
  • network-rgb-h264-sender / network-rgb-h264-receiver and rtsp-rgb-h264-sender / rtsp-h264-receiver: UDP and RTSP streaming (Linux and Jetson only)

The Jetson folder additionally provides the zedxone-* scripts for the ZED X One cameras.

Jetson low-latency, zero-copy pipelines

The scripts/jetson folder contains pipelines optimized for hardware-accelerated streaming on NVIDIA® Jetson™:

  • Zero-copy NV12: with ZED SDK v5.2 or newer and a GMSL2 camera, zedsrc and zedxonesrc can deliver NV12 buffers directly to the hardware encoder, with no CPU copy and minimal latency
  • SRT, RTSP and UDP streaming: low-latency streaming scripts with automatic camera detection
  • DeepStream integration: object detection pipelines using NVIDIA DeepStream

The scripts detect the camera type (GMSL2 or USB3) and the ZED SDK version to select the optimal pipeline.

Use cases

The following is a list of use case examples, each linked to its relevant documentation page: