Skip to navigation

Sensors Time Synchronization

Open in ClaudeOpen in ChatGPT

Accurate time synchronization is required for reliable integration of data from multiple sensor sources. The ZED SDK timestamps the images and the sensor data of a camera on the same host clock, with nanosecond resolution. These timestamps can be used to:

  • Integrate multi-sensor data from a single camera such as IMU and stereo images.
  • Fuse data with external sensor sources such as GPS or LiDAR.

This page explains how to align IMU samples with the images returned by grab(), which time reference to use for each need, and the mistakes that lead to misaligned data in a fusion pipeline.

Image and Sensor Timestamps

Since images and sensor samples share the same clock, their timestamps can be compared directly:

  • The timestamp of the last grabbed frame is returned by getTimestamp(TIME_REFERENCE::IMAGE).
  • The timestamp of an IMU sample is stored in sensors_data.imu.timestamp. It is the time of the measurement, not the time your application read it.

By default, timestamps are in Epoch time. Since ZED SDK 5.3, Camera::setTimestampClock() can select a monotonic clock instead, which is not affected by NTP or PTP adjustments.

The TIME_REFERENCE enum selects which instant a function refers to:

ValuegetTimestamp() returnsgetSensorsData() returns
IMAGEThe timestamp of the last grabbed frameThe IMU sample closest to the last grabbed frame
CURRENTThe current timeThe most recent sample received from the sensors
IMAGE_CENTER_OF_EXPOSUREThe middle of the exposure of the last grabbed frame, see belowNot supported, returns ERROR_CODE::INVALID_FUNCTION_PARAMETERS

Center of Exposure

The IMAGE timestamp is taken at the start of the sensor readout, which is the moment the exposure of the frame ends. A sensor that timestamps the instant it measures, such as an IMU, a LiDAR or a robot joint encoder, is therefore compared with an image whose light was collected earlier.

Since ZED SDK 5.5, getTimestamp(TIME_REFERENCE::IMAGE_CENTER_OF_EXPOSURE) returns the middle of the exposure window, which is half an exposure earlier than IMAGE. With an exposure time of 11.7 ms, for example, it is 5.9 ms earlier. Use it as the frame time when you fuse images with such sensors, especially when the exposure time is long compared to the motion you want to capture.

The center of exposure is only available on the ZED X, ZED X Mini, ZED X One GS and ZED X One 4K, live or from an SVO file recorded with one of these cameras and ZED SDK 5.1 or later. On other inputs (USB cameras, HDR models, files without per-frame exposure), getTimestamp() returns 0 and logs a warning: always check for 0 and fall back to IMAGE.

  • On the rolling shutter ZED X One 4K, the value refers to the first row of the image. The following rows are exposed progressively later.
  • For SVO files recorded with a ZED SDK older than 5.5, the value is computed with the exposure of a later frame. See the ZED SDK 5.5 release notes.
// Time of the frame that grab() just returned, at the center of its exposure when available
uint64_t frame_ns = zed.getTimestamp(TIME_REFERENCE::IMAGE_CENTER_OF_EXPOSURE).getNanoseconds();
if (frame_ns == 0) // Not available on this camera or input
frame_ns = zed.getTimestamp(TIME_REFERENCE::IMAGE).getNanoseconds();

Getting Time Synced Sensors Data

The IMU and the image sensor run at different rates: the IMU of a ZED 2i, for example, runs at 400 Hz, while its images are captured at up to 100 FPS. Read the IMU rate of your camera from sensors_configuration.accelerometer_parameters.sampling_rate in the camera information instead of hard-coding it.

The ZED SDK offers three ways to read IMU data, described below.

TIME_REFERENCE::IMAGE and TIME_REFERENCE::CURRENT

The function getSensorsData(sensors_data, TIME_REFERENCE) fills a SensorsData structure with a single sample of each sensor, as explained in Using the Sensors API. The time reference selects which IMU sample is returned:

  • TIME_REFERENCE::IMAGE returns the IMU sample closest to the timestamp of the last grabbed frame. It can be slightly before or slightly after the frame, usually by a few milliseconds.
  • TIME_REFERENCE::CURRENT returns the most recent IMU sample available.

The Camera-IMU synchronization schema below illustrates the values returned with these two settings.

Camera-IMU synchronization schema

In the example above, we call grab() and retrieveImage() to retrieve the most recent image frame available from the camera. Then we call one of the two functions below:

  • getSensorsData(sensors_data, TIME_REFERENCE::IMAGE) returns IMU sample “1”, captured near image frame “I0”. Calling the function again before a new frame is available returns the same IMU sample “1”.

  • getSensorsData(sensors_data, TIME_REFERENCE::CURRENT) returns the most recent IMU sample. Calling the function twice returns IMU samples “2” and “4”, independently of the grab() calls and of the image timestamps.

getSensorsDataBatch()

Since ZED SDK 5.1, getSensorsDataBatch() returns, as a list of SensorsData sorted oldest first, every IMU sample not yet returned whose timestamp is not newer than the frame that grab() just returned. Call it once after each grab() to process the IMU at its full rate without running a separate polling thread:

  • The list is filled once per grab(). Calling the function again before the next grab() returns an empty list.
  • The first call also returns the samples received before the first frame.
  • The samples received while your application does not call grab(), for instance during a long processing step, are returned by the next call. With a live camera, the SDK keeps up to 2.5 seconds of samples with the ZED 2 and ZED 2i, and 0.5 seconds with the other cameras: older samples are lost.
  • On GMSL cameras (ZED X family), the IMU data reaches the host in bursts. A sample can arrive after the frame that follows it was grabbed: it is then returned with the next frame, with a timestamp a few milliseconds older than the previous frame. Assign the samples to frames with their timestamps, not with the call that returned them, as shown in the full-rate example.
  • In C++, getSensorsDataBatch() appends the samples to the vector you pass without clearing it. Call clear() before each call, or declare a new vector.
  • With an SVO file recorded in the legacy SVO 1 format, which does not contain the full-rate IMU data, getSensorsDataBatch() and TIME_REFERENCE::CURRENT return ERROR_CODE::INVALID_FUNCTION_PARAMETERS. The same applies to a network stream that does not carry the full-rate IMU data.

Which One to Use

NeedUse
One IMU sample per frame, for example to tag each image with the camera orientation or the gravity directiongetSensorsData() with TIME_REFERENCE::IMAGE after each grab()
Every IMU sample between two frames, in step with the images, for visual-inertial odometry, an EKF or IMU preintegrationgetSensorsDataBatch() after each grab(), with the frame timestamps
The IMU at full rate with the lowest latency, independently of grab(), for a control loop or to publish the IMU as a separate streamgetSensorsData() with TIME_REFERENCE::CURRENT in a dedicated thread, see Retrieve New Sensor Data
Reading an SVO fileTIME_REFERENCE::IMAGE or getSensorsDataBatch()

Since getSensorsDataBatch() returns the samples once per frame, a sample can wait up to one frame period, plus the time needed to transfer and process the image, before your application reads it. This is why the lowest latency requires TIME_REFERENCE::CURRENT in a dedicated thread.

Common Mistakes

The following mistakes are the most frequent causes of misaligned IMU and image data.

Calling getSensorsData() with TIME_REFERENCE::CURRENT right after grab() to get the IMU sample of the frame. grab() returns after the image is transferred and processed, so the most recent IMU sample is not the one closest to the frame. In our tests, it was 26 to 64 ms newer than the image with a ZED 2i, and from 14 ms older to 64 ms newer with a ZED X Mini. During SVO playback, CURRENT does not follow the time of the frame either. Use TIME_REFERENCE::IMAGE or getSensorsDataBatch().

Polling TIME_REFERENCE::CURRENT at the IMU rate. Each call returns only the newest sample, and GMSL cameras deliver the IMU in bursts: a loop running at the IMU rate misses samples. Poll several times faster than the IMU rate and keep a sample only when its timestamp is newer than the last one, or use getSensorsDataBatch() if you do not need the lowest latency.

Reading TIME_REFERENCE::IMAGE data or the batch from a thread other than the one calling grab(). Both change at each grab(), so a call made while another frame is being grabbed can return the data of the next frame. The batch can also be read only once per frame: if two threads read it, one of them gets an empty list. Read them right after grab(), in the same thread.

Comparing ZED timestamps with another clock. The image and IMU timestamps use the clock selected with Camera::setTimestampClock(). To get the current time on the same clock, use sl::getCurrentTimeStamp() (sl.get_current_timestamp() in Python, zed.GetCurrentTimeStamp() in C#) instead of the clock of your language, such as std::chrono::system_clock or time.time(), which differs from the ZED timestamps when a monotonic clock is selected.

Example: One IMU Sample per Frame

For each grabbed frame, this example reads the IMU sample closest to the image with TIME_REFERENCE::IMAGE. It prints the offset between the two timestamps, the orientation of the camera and its acceleration.

#include <sl/Camera.hpp>
#include <iostream>
using namespace sl;
int main() {
Camera zed;
InitParameters init_parameters;
init_parameters.depth_mode = DEPTH_MODE::NONE; // Depth is not needed for this example
if (zed.open(init_parameters) > ERROR_CODE::SUCCESS) return EXIT_FAILURE;
SensorsData sensors_data;
for (int i = 0; i < 300; i++) {
if (zed.grab() > ERROR_CODE::SUCCESS) continue;
// Timestamp of the frame that grab() just returned
Timestamp image_ts = zed.getTimestamp(TIME_REFERENCE::IMAGE);
// IMU sample closest to that frame
if (zed.getSensorsData(sensors_data, TIME_REFERENCE::IMAGE) > ERROR_CODE::SUCCESS) continue;
const SensorsData::IMUData& imu = sensors_data.imu;
// Offset between the IMU sample and the image, in microseconds
long long offset_us = (static_cast<long long>(imu.timestamp.getNanoseconds())
- static_cast<long long>(image_ts.getNanoseconds())) / 1000;
std::cout << "Image " << image_ts.getNanoseconds() << " ns | IMU offset " << offset_us << " us"
<< " | Orientation " << imu.pose.getOrientation()
<< " | Acceleration " << imu.linear_acceleration << " m/s2\n";
}
zed.close();
return EXIT_SUCCESS;
}

Example: Every IMU Sample Between Two Frames

This example reads every IMU sample with getSensorsDataBatch() and assigns each one to the interval between two frames, as needed by a visual-inertial filter: the filter is propagated with the IMU samples up to the time of a frame, then updated with that frame.

Since an IMU sample can arrive with the next frame, the example processes each frame one grab() later, once all of its samples have been received. It uses the center of exposure as the frame time when it is available. If the latency of one frame is not acceptable, process the frame right away and handle the late samples in your filter.

#include <sl/Camera.hpp>
#include <deque>
#include <iostream>
using namespace sl;
// Time of the frame that grab() just returned. Prefer the center of exposure
// (ZED X, ZED X Mini, ZED X One GS, ZED X One 4K) and fall back to IMAGE when it is not available.
uint64_t getFrameTime(Camera& zed) {
uint64_t t = zed.getTimestamp(TIME_REFERENCE::IMAGE_CENTER_OF_EXPOSURE).getNanoseconds();
return t != 0 ? t : zed.getTimestamp(TIME_REFERENCE::IMAGE).getNanoseconds();
}
int main() {
Camera zed;
InitParameters init_parameters;
init_parameters.depth_mode = DEPTH_MODE::NONE; // Depth is not needed for this example
if (zed.open(init_parameters) > ERROR_CODE::SUCCESS) return EXIT_FAILURE;
std::vector<SensorsData> batch;
std::deque<SensorsData::IMUData> imu_queue; // IMU samples not yet assigned to a frame
uint64_t last_imu_ns = 0; // newest IMU sample already queued
uint64_t prev_frame_ns = 0; // frame waiting for its IMU samples
for (int i = 0; i < 300; i++) {
if (zed.grab() > ERROR_CODE::SUCCESS) continue;
uint64_t frame_ns = getFrameTime(zed);
// IMU samples up to this frame that were not returned yet, oldest first.
// getSensorsDataBatch() appends to the vector: clear it first.
batch.clear();
if (zed.getSensorsDataBatch(batch) > ERROR_CODE::SUCCESS) continue;
for (const auto& data : batch) {
uint64_t t = data.imu.timestamp.getNanoseconds();
if (t <= last_imu_ns) continue; // Skip a sample already received
imu_queue.push_back(data.imu);
last_imu_ns = t;
}
// Process the previous frame: by now, its IMU samples have all been received
if (prev_frame_ns != 0) {
int count = 0;
while (!imu_queue.empty() && imu_queue.front().timestamp.getNanoseconds() <= prev_frame_ns) {
const SensorsData::IMUData& imu = imu_queue.front();
// Feed imu.angular_velocity and imu.linear_acceleration to your filter here
imu_queue.pop_front();
count++;
}
// Then update your filter with the frame taken at prev_frame_ns
std::cout << "Frame " << prev_frame_ns << " ns: " << count << " IMU samples since the previous frame\n";
}
prev_frame_ns = frame_ns;
}
zed.close();
return EXIT_SUCCESS;
}

Using the API

To learn how to read the data of each sensor, read the Using the Sensors API section.