ZED Topic Benchmark Tool
The ZED Benchmark tool measures the real performance of the topics published by the ZED ROS 2 nodes: how many messages per second actually reach a subscriber, how much bandwidth they consume, how long they take to arrive, and how much CPU the transport costs. It is the recommended way to verify that a machine, a network, and a DDS configuration are able to deliver the camera data at the rate the node is configured to publish it, and to measure what Composition and Intra Process Communication actually buy.
The tool is distributed in the tools/benchmark folder of the zed-ros2-examples repository.
For each received message the following information is available:
- Topic type
- Instantaneous frequency
- Windowed average frequency
- Message size
- Instantaneous topic bandwidth
- Windowed average topic bandwidth
- End-to-end latency, from the publisher-side
header.stampto the arrival in the benchmark callback - CPU time consumed by the subscribing process
The values are printed on the console in real time and published on a dedicated topic, so they can be plotted with tools like PlotJuggler or rqt_plot. When the test ends, a final report with the aggregated statistics of the whole session is printed and, optionally, saved to a file.
By default the benchmark subscribes to the topic with a generic subscription, so it works with any ROS 2 topic, not only with the topics published by the ZED nodes. Measuring latency, and measuring Intra Process Communication, requires instead a typed subscription, which supports a specific list of message types. See Measuring Intra Process Communication.
The tool is available for ROS 2 Humble Hawksbill, Jazzy Jalisco, Lyrical Luth, and newer distributions. It is not available for ROS 2 Foxy and older releases, because they lack the generic subscription API required to subscribe to topics of an arbitrary type. On those distributions the three packages are still registered in the ament index, so the workspace build succeeds, but nothing is produced: no message, no component library, and no zed_topic_benchmark executable. A warning is printed by CMake during the build.
The packages
Installation
The benchmark packages are built together with the other packages of the zed-ros2-examples repository:
If you only need the benchmark tool, you can restrict the build to its packages:
The benchmark packages do not require CUDA or the ZED SDK, so they can also be installed on a remote workstation that only receives the data over the network. This is the best way to measure what a client machine really receives.
Optional dependencies
Some capabilities are compiled in only if the corresponding package is found at build time. Everything is optional: what is missing is simply not available, and the build never fails because of it. CMake prints which tiers it enabled, for example:
A message type that was not compiled in falls back to a generic subscription, with a warning that says so.
Usage
Start the ZED node, then open a new terminal console and run the benchmark node on the topic you want to measure:
For example:
The node prints the benchmark configuration, waits for the topic to be advertised (it polls the graph every 500 msec), and then starts measuring:
The lines starting with # are the live statistics line, which is updated in place: only one such line is visible while the test runs. The counter is the number of processed messages, and frequency and bandwidth are reported as instant/average. The layout is fixed-width and stays within 80 columns so that it does not wrap on a default terminal.
The example above was captured with a ZED 2i publishing rectified color images at HD720 and 60 FPS.
By default the benchmark runs until you stop it with Ctrl+C. You can also limit it by time or by number of messages, and save the final report to a file.
Run for 30 seconds, then stop and print the report:
Run until 1000 messages are received, then stop:
Run for 60 seconds and also save the report to a log file:
Parameters
All the parameters are read-only: they must be set when the node is started and cannot be modified at runtime.
When both test_duration_sec and test_sample_count are set, the test stops as soon as the first of the two limits is reached.
The value of avg_win_size defines how reactive the average is. A large window (e.g. 500) gives a very stable value, useful to characterize a steady stream; a small window (e.g. 30) reacts quickly and makes throughput drops easier to spot.
Final report
The report is generated when a test limit is reached or when the node is stopped with Ctrl+C. It is printed to the console and, if log_file_path is set, written to that file:
The report contains:
- Topic name and topic type as discovered on the ROS 2 graph.
- Subscription: the subscription path actually used,
generic (runtime-typed),typed (<message type>), ortype-adapted (zero-copy, sl::Mat by pointer). A topic that was never seen reportsnone - the topic was never seen. - Delivery path: how the messages were delivered, see What the report claims.
- Size semantics: whether the sizes are
serialized wire bytes(generic path) ormessage content bytes(typed paths). The two are not comparable, so a stored report can no longer be misread. - Subscriber QoS: the QoS the middleware actually granted, read back from the subscription rather than echoing what was requested.
- Stop reason:
test completed (duration reached),test completed (sample count reached), orinterrupted by the user (Ctrl+C). - Test duration: time between the first and the last received message.
- Messages received and total data received during the test.
- Mean / min / max of frequency, message size, bandwidth, and latency.
- Process CPU: CPU time consumed by the process the benchmark runs in, and its percentage of one core. See Reading the CPU figure.
If no message is received at all, the report says so explicitly instead of printing meaningless statistics, while still reporting the subscription path and the QoS so the cause is visible.
The same topic measured by the component composed in the camera container, on the type-adapted zero-copy path:
Both reports come from the same ZED 2i at HD720 and 60 FPS. Note that the composed Process CPU is much larger simply because that process is the ZED node.
Understanding the statistics
- The instantaneous frequency and bandwidth are computed from the last inter-arrival interval, measured with a monotonic clock (
steady_clock), so the measurement is not affected by host clock adjustments (NTP/PTP). - The average frequency and bandwidth are computed on the message periods and sizes, i.e. as
1/mean(Δt)andmean(size)/mean(Δt). This is the unbiased windowed rate also used byros2 topic hz. Averaging the instantaneous rates directly would systematically overestimate the real frequency, increasingly so on jittery topics. - In the final report, the mean frequency and bandwidth are computed over the whole test (total messages over total time and total data over total time).
- The report min/max are tracked on the windowed average rate, and only after the averaging window has been filled. Tracking them on the raw instantaneous rate would report meaningless extremes caused by a single very short inter-arrival interval, e.g. two messages delivered back-to-back by the executor or a publisher burst. If the test is too short for the window to fill, min and max fall back to the overall mean values.
- The message size is printed with adaptive units (B / KB / MB / GB), so small messages such as IMU samples are not rounded to
0.00 MB. On the generic path it is the exact size of the serialized message, i.e. the wire bytes; on the typed paths it is the size of the message content, without the CDR framing. The report always states which of the two it is. - Bandwidth values are expressed in Mbps computed on binary multiples (1 Mb = 1024 × 1024 bits). On the intra-process path no bytes are transported at all, so the bandwidth there is a notional payload figure and must not be compared with an inter-process one.
- The latency is the time between the publisher-side
header.stampand the arrival in the benchmark callback. It requires a publisher that fills the stamp and is available only on the typed paths, because a generic subscription receives an opaque serialized buffer. See Reading the latency correctly, it is easy to misinterpret on ZED topics. - The very first received message only initializes the time reference: no frequency can be derived from a single message.
Frequencies measured by the benchmark tool can be lower than the publishing frequencies reported by the node diagnostic. The diagnostic measures the rate at which the node publishes, while the benchmark measures the rate at which a subscriber receives. A difference between the two means that messages are being lost or delayed by the middleware, by the network, or by the receiving machine; see DDS and Network Tuning.
Plotting the statistics
The node publishes the statistics on the topic <name_of_the_benchmarked_topic>_stats, e.g. /zed/zed_node/rgb/color/rect/image_stats, so they can be recorded with ros2 bag or plotted in real time.
The message is a custom type, zed_topic_benchmark_interfaces/msg/BenchmarkStatsStamped, defined as:
The two latency fields are 0.0 on the generic path, which has no directly usable publisher timestamp.
For example, to plot the average frequency and latency with rqt_plot:
QoS
The subscriber QoS is fully configurable through four parameters, and they are effective on every subscription path:
The defaults reproduce the historical behavior of the tool. A Best Effort subscriber is compatible with both Reliable and Best Effort publishers, so the benchmark connects out of the box to any topic.
An unrecognized value is replaced by the default and reported as a warning, never applied silently, and a qos.depth below 1 is clamped to 1, also with a warning. Every report states the QoS the middleware actually granted, read back from the subscription instead of echoing the request, e.g. Subscriber QoS: Reliable, Volatile, KEEP_LAST, depth 1. You can cross-check it from another terminal while the benchmark runs:
The default shallow queue (depth 1) is intentional: it makes the tool measure the data that is actually delivered in time, instead of hiding losses behind a large buffer. Increase qos.depth when you want to measure how much data arrives in total rather than how much arrives on time.
An incompatible QoS request yields no messages at all. DDS delivers only when the subscriber’s request is no stronger than what the publisher offers. The ZED nodes publish their topics Reliable, Volatile, KEEP_LAST (depth 10) by default, which means:
qos.reliability:=reliableworks on ZED topics. It yields nothing only against aBest Effortpublisher, so keep the default when benchmarking a topic from another node whose reliability you have not checked.qos.durability:=transient_localis incompatible with aVolatilepublisher, so it delivers nothing on ZED topics. The ZED node logsrequesting incompatible QoS ... Last incompatible policy: DURABILITY_QOS_POLICY.
The benchmark logs the caveat up front whenever reliable or transient_local is selected, and the report pairs the requested QoS with No message received, so the cause is visible rather than mysterious. Use ros2 topic info -v <topic> to check the publisher’s side.
Why not the standard qos_overrides?
The usual ROS 2 mechanism, -p qos_overrides.<topic>.subscription.reliability:=reliable, does not work on the default generic path: rclcpp::create_generic_subscription() forwards the subscription options but never calls declare_qos_parameters(), so no qos_overrides.* parameter is ever declared and the requested policy never reaches the endpoint. This is true on the humble, jazzy, lyrical and rolling branches of rclcpp alike.
The qos.* parameters above avoid the problem entirely, because they set the QoS argument of the subscription, which every path honors. If you pass a qos_overrides.* anyway, the benchmark warns that it is being ignored and points you at qos.* instead of dropping it in silence:
On the typed path qos_overrides.* does work, because there rclcpp does declare the parameters, and it is applied on top of qos.*. Prefer qos.* regardless: it behaves identically on both paths.
The same limitation affects the ROS 2 CLI: ros2 topic hz and ros2 topic bw expose no QoS options and hard-code the sensor-data profile. ros2 topic echo is the only verb that accepts --qos-* options.
Measuring Intra Process Communication
The zed_topic_benchmark_component package provides the ROS 2 component stereolabs::TopicBenchmarkComponent, so the benchmark can be loaded into the same component container as the ZED node instead of running in its own process. Composing it is what makes Intra Process Communication measurable, but it is not sufficient on its own.
Why a generic subscription can never measure IPC
Subscribing to a topic whose type is only discovered at runtime requires an rclcpp::GenericSubscription. In rclcpp, a subscription is registered with the IntraProcessManager only from the constructor of the templated rclcpp::Subscription<T>, and GenericSubscription derives directly from SubscriptionBase and never does that. This is the case on every distribution from Humble to Rolling, and there is no serialized intra-process path to opt into either.
A generic subscription therefore always receives through the middleware, even inside the publisher’s own container with use_intra_process_comms:=true. This is genuinely confusing to observe, because the ZED node can correctly log [IPC type-adapted zero-copy] while the benchmark measures the inter-process path: with IPC enabled the publisher feeds both paths, and only a real intra-process subscriber benefits.
The subscription_mode parameter
auto keeps an ordinary separate-process run on the historical generic subscription, so its wire-accurate bandwidth figures are unchanged, while a composed run with IPC enabled automatically gets a subscription that can actually use it. If a typed subscription is requested for an unsupported type, the benchmark falls back to generic with a warning rather than silently measuring something else.
Supported message types
image_transport and point_cloud_transport publish the same image or cloud once per transport plugin, each on its own sub-topic and with its own message type. All of them can be measured on the typed path:
For a compressed cloud the compressed payload is reported, not the uncompressed geometry, which would overstate the transported volume by the whole compression ratio. Some of these types require an optional dependency at build time.
Two tiers of typed delivery
Not all intra-process delivery is zero-copy, and the difference is large:
- Plain typed subscription: skips serialization and the middleware entirely, but rclcpp still copies the message into the subscription’s buffer. When the publisher is type-adapted, rclcpp first calls
convert_to_ros_message()and then copies the result, i.e. two full copies of the image. - ZED type-adapted subscription: built on the very same
TypeAdapter<StampedSlMat, sensor_msgs::msg::Image>the ZED node publishes with, so the publisher’ssl::Matarrives by pointer, with no serialization, no conversion and no copy. This is genuine zero-copy, and it is selected automatically forsensor_msgs/msg/Imagetopics when intra-process comms are enabled and the optionalzed_componentsdependency was available at build time.
Composing the benchmark
Start the ZED node with IPC enabled:
then load the benchmark component into the ZED container:
You can verify that the benchmark node is running in the same process as the ZED node with ros2 component list:
Set use_ros_log:=true when the component runs inside a container: the in-place console update of the live statistics line is not usable when the output is multiplexed with the logs of the other nodes.
When the benchmark component reaches a test_duration_sec or test_sample_count limit, it requests the shutdown of the process it is running in. In a shared container this stops the whole container, including the ZED node and any other composed node. The reports are still generated before the shutdown. To benchmark composed nodes for an arbitrary time, leave both limits at their default (infinite) and stop the container with Ctrl+C.
What to compare, and it is not bandwidth
On the intra-process path no bytes are transported, so a bandwidth figure there is notional payload throughput and is not comparable with an inter-process one. The frequency is set by the publisher and barely moves either. The metrics that show the gain are latency and CPU.
Measured on a ZED 2i, rgb/color/rect/image (3.52 MB per frame, about 1.7 Gbps) at 60 FPS with NEURAL_LIGHT depth running, on an RTX 4070 desktop, both runs with subscription_mode:=typed:
On that desktop, zero-copy saves about 2 ms of latency and 4 points of CPU on a 1.7 Gbps stream. The gain looks modest because both figures are dominated by work that is not the transport: 70 of those CPU points are capture plus depth, and most of the latency is the camera pipeline. Isolating the transport with a synthetic type-adapted publisher that stamps at publish time and does no camera work, the same code reports 0.02 ms against 3.32 ms, and 0.34% against 2.04% of one core.
The gain is far larger on an embedded target, where the CPU is weaker relative to the data rate. On a Jetson AGX Orin (JetPack 6, L4T R36.5) with a GMSL2 ZED X Mini at HD1200, rgb/color/rect/image is 8.79 MB at 30 FPS, about 2.1 Gbps:
Zero-copy costs 2.6 times less CPU for the transport there, and saves about 6 ms of latency. With four GMSL2 cameras (two ZED X Mini and two ZED X) in a single container, all four confirming zero-copy at 8.79 MB per frame, roughly 8 Gbps aggregate:
Composed, the cameras nearly hold their nominal 30 FPS. As separate processes the middleware cannot keep up, every camera drops frames, and it costs 78% more CPU.
Reading the latency correctly
The ZED node stamps images and clouds with the frame acquisition time (sl::TIME_REFERENCE::IMAGE), not the publish time. The reported latency therefore covers the whole pipeline, i.e. capture, transfer, SDK retrieve, rectification and depth, publish and deliver, and the transport is only a small part of it. On a ZED 2i at 60 FPS the rectified RGB image reports about 18 to 20 ms in both modes, of which only around 2 ms is the transport. Compare the difference between the two modes, not the absolute value.
This applies to every topic of the node, camera_info included: they are all stamped from the frame timestamp, so none of them isolates the transport on its own. A compressing transport adds its own work on top of that, which is why depth/depth_registered/compressedDepth reports 53.9 ms on a ZED 2i and 253 ms on a Jetson AGX Orin: the stamp predates the compression that the plugin then performs.
To time the transport alone, set the wrapper’s use_pub_timestamps parameter to true, which stamps at publish time instead. That is the only reliable way to obtain a transport-only figure from this tool.
Latency is reported as not available on the generic path, which receives a serialized buffer with no directly usable timestamp.
Reading the CPU figure
The report’s Process CPU covers the whole process. Composed, that process is the ZED node, so the figure also includes capture, depth and publishing, and it is therefore not comparable with the separate-process figure, which covers the benchmark alone. On a ZED 2i this reads 6.2% separate against 89.4% composed, which naively suggests that IPC is far worse, while the totals say the opposite.
The fair comparison is the total across every process involved, which is what zed_check_ros2_config.sh reports in its CPU_TOT column, alongside a CPU_IDLE baseline measured with no subscriber at all, so the transport cost of a mode is CPU_TOT - CPU_IDLE.
What the report claims, and what it does not
A typed subscription on a node with IPC enabled takes the intra-process path only for publishers that live in the same process, and that cannot be checked from inside a callback. The report therefore distinguishes three cases:
intra-process, zero-copy - CONFIRMED: only the type-adapted path can prove this, because the custom C++ type it receives has no wire representation and so cannot have come through the middleware.intra-process capable, NOT confirmed: typed subscription with IPC enabled, but delivery is not verifiable per message. Compare the latency against an inter-process run to see which path you got.inter-process (middleware): with the reason, including the case where a generic subscription makes the intra-process path impossible by construction.
Benchmarking a multi-camera setup
The zed_test_ipc.launch.py launch file of the zed_topic_benchmark package starts a multi-camera setup and one benchmark node per camera, either composed in the camera container using IPC or as separate processes:
The available launch arguments are:
For each camera the launch file benchmarks the topic /zed_multi/<cam_name><topic_name> with a node named benchmark_<index> in the zed_multi namespace, and all the benchmark nodes use the ROS logging system (use_ros_log is forced to true).
Both branches now share a single parameter set, so a use_ipc:=True run and a use_ipc:=False run are directly comparable. This matters for avg_win_size in particular: the report min/max are tracked on the windowed average, so an asymmetric window alone would make the two runs incomparable.
As explained in Composing the benchmark, with use_ipc:=True all the benchmark nodes share the camera container: a finite test_duration_sec/test_sample_count stops the whole container as soon as the first node reaches its limit. Keep the limits at their defaults and stop the launch with Ctrl+C to benchmark a composed multi-camera setup for an arbitrary time.
Testing your ROS 2 configuration
The zed_topic_benchmark package ships a helper script, zed_check_ros2_config.sh, that automates a fixed set of benchmarks against a real ZED camera and prints a report for each one. Its purpose is to verify that your ROS 2, DDS, and system configuration is able to deliver the camera data at the expected rate and bandwidth, a quick way to validate a new setup or to investigate performance problems.
The script measures each of the following topics twice, once with the benchmark in a separate process (interprocess) and once composed in the camera container with intra-process comms enabled (ipc):
The RGB image is measured with depth disabled (NONE) to isolate the image throughput, while depth and point cloud are measured with the default depth mode (NEURAL_LIGHT).
Both runs use subscription_mode:=typed, so the size and latency accounting is identical and the two really are comparable, and the ZED node keeps enable_ipc:=true in both, so the publisher is the same and the only variable is where the subscriber lives.
For each of the six tests, the script starts the ZED node with the required depth.depth_mode, waits for the topic to be advertised, waits WARMUP seconds for the node to reach steady state, measures a no-subscriber CPU baseline, runs the measurement, saves the report as report_<mode>_<topic>.txt, and then stops the node to release the camera.
The script drops the ROS 2 CLI daemon cache at start-up, and retries the component load once after dropping it again. A stale graph cache can otherwise make ros2 component load fail with “Unable to find container node”, or make ros2 topic list return an incomplete graph so that every test is skipped as “topic not available”.
Usage
For example, for a ZED 2i with 20 seconds of measurement per test:
The first argument is the camera model (zed, zedm, zed2, zed2i, zedx, zedxm, …) and defaults to zed2i; the second one is the duration in seconds of each measurement and defaults to 15. The behavior can be further tuned with environment variables:
The script starts and stops the ZED node itself, so no other node must be using the camera when you run it. Make sure that no ZED node is already running and that the camera is free.
The script must be run in a terminal where the ROS 2 and workspace environments are sourced (it checks that the ros2 command is available). Plan for a few minutes of execution: six tests, each including the start-up of the ZED node, the measurement, and the settle time.
Reading the results
A full report is saved for each test in the output directory, together with the log of the ZED node, useful if a test fails or if a topic never appears. At the end, a summary table is printed:
The summary layout is the following. The values are placeholders: the actual numbers depend entirely on your camera, resolution, middleware and machine.
CPU_TOTis the total across every process involved, i.e. the ZED node plus the benchmark forinterprocess, and the ZED node alone foripc, where the benchmark runs inside it. This is the only CPU figure that is comparable between the two modes, see Reading the CPU figure.CPU_IDLEis the ZED node with no subscriber at all, so the transport cost of a mode isCPU_TOT - CPU_IDLE. ACPU_IDLEaboveCPU_TOTmeans the baseline was sampled while the node was still warming up: re-run with a largerWARMUP.- A
NO DATArow means that no report was produced for that test: the topic was not published withinTOPIC_TIMEOUT, or the ZED node failed to start. Check the correspondingzed_<mode>_<topic>.logfile in the output directory.
Read the table like this:
- Compare the measured frequencies with the values configured through
general.pub_frame_rateanddepth.point_cloud_freq(see Node Frequency Tuning). Values well below them mean the data is not being delivered. - Expect the two
FREQvalues of a topic to be close, because the publisher sets the rate. - Compare
LATENCYandCPU_TOTbetween the two modes: those are what the intra-process path changes. Do not compare bandwidth, which is not reported here for exactly that reason. For measured examples see What to compare.
When the results are not as expected
If the measured frequencies are significantly lower than the configured publishing frequencies, or the interprocess latency is high, your ROS 2 middleware or your system is likely not tuned for high-throughput data. The following pages describe how to fix it:
- DDS and Network Tuning for ROS 2: the DDS and kernel settings required to reliably transport large messages such as images and point clouds.
- Using Zenoh as Middleware: an alternative middleware, recommended for distributed and Wi-Fi based setups.
- Node Frequency Tuning: how to configure the publishing frequency of each module to match the available computational power and bandwidth.
- Composition and IPC: how to remove the middleware overhead entirely when publisher and subscriber run on the same machine.
Testing the packages
The statistics engine, i.e. the sliding-window averages and the frequency/bandwidth math, is covered by C++ unit tests; the end-to-end behavior, i.e. report generation and self-termination on a sample-count limit, is covered by a launch_testing integration test that runs the node against a synthetic publisher.
Always run colcon from the root of your ROS 2 workspace (the folder containing src/, build/, and install/). Running it from another directory makes colcon scan that directory recursively, which can pull in unrelated setup.py files and abort with package-identification errors.
The integration test can also be run directly with launch_test: