> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.stereolabs.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.stereolabs.com/_mcp/server. # Tutorial - Object Detection with Isaac™ ROS This tutorial explains how to detect objects in the images of a ZED camera with the [Isaac™ ROS 5 object detection](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_object_detection/index.html) packages, accelerated with TensorRT. The example supports two models: * **RT-DETR** (default): the NVIDIA® [SyntheticaDETR](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/models/synthetica_detr) `sdetr_grasp` model, trained on synthetic data to detect objects that a robot arm can grasp. * **YOLOv8**: an [Ultralytics YOLOv8](https://docs.ultralytics.com/models/yolov8/) model, trained by default on the 80 classes of the COCO dataset (people, vehicles, animals, everyday objects). The example is designed for the monocular **ZED X One** cameras, which do not provide depth and cannot use the Object Detection module of the ZED SDK. It works with all the ZED cameras: with stereo cameras, the objects are detected in the left image. The image stays in GPU memory from the ZED SDK to the TensorRT inference, thanks to the [CUDA zero-copy transport](/docs/integrations/isaac-ros/cuda-zero-copy/). Read that page first for an introduction to how the transport works. ## The detection pipeline The example chains the ZED node, the Isaac™ ROS image and tensor processing nodes, the TensorRT node, and the decoder of the selected model. A small Python node then maps the detections back to the ZED image. ![The ZED Isaac ROS object detection pipeline, and where the CUDA zero-copy transport applies](/_fern-files/stereolabs.docs.buildwithfern.com/3c906b4c69f3600202ed5629faad5ba86774adb4debff8c28fe0519a1551920a/docs/software-integration/isaac-ros/images/zed-isaac-ros-object-detection-pipeline.svg) *Figure 1: the object detection pipeline. Green arrows carry data in GPU memory, orange arrows mark the copies from GPU to CPU memory.* The image is converted to the input tensor of the network in several steps, all on the GPU: 1. **Resize**: scales the ZED image to fit in 640x640, keeping its aspect ratio. 2. **Pad**: adds black pixels on the bottom and on the right side, to obtain a 640x640 image. 3. **Format converter**: converts the `bgr8` image to `rgb8`. 4. **Image to tensor**: converts the pixels to `float32` values, in \[0, 1] for YOLOv8 and in \[0, 255] for RT-DETR. 5. **Interleaved to planar** and **Reshape**: reorder the tensor in the `NCHW` layout expected by the network. 6. **RT-DETR preprocessor** (RT-DETR only): adds the size of the input image, a second input of the RT-DETR network. The **TensorRT** node runs the inference, and the **decoder** of the selected model converts the output tensors into a `vision_msgs/Detection2DArray` message, in the coordinates of the 640x640 network image. The **detection rescaler** maps the detections back to the coordinates of the ZED image, and publishes them with the Foxglove annotations. ### Where the CUDA zero-copy transport makes the difference The numbers below match the badges of Figure 1. 1. **From the ZED node to the Isaac™ ROS nodes** (cross-process). The ZED node and the Isaac™ ROS nodes run in two different processes. With the `cuda` buffer backend, the middleware sends only a small descriptor of the GPU memory where the ZED SDK wrote the image, and the Resize node reads the same memory. Without it, the ZED node copies every image to CPU memory, the middleware serializes and transfers the full image, and the Resize node uploads it back to the GPU. 2. **Between the Isaac™ ROS nodes** (intra-process). The Isaac™ ROS nodes run in the same container and exchange their messages with rclcpp Intra-Process Communication: each node passes a pointer to its output message, whose `rosidl::Buffer` stays in GPU memory. This is where the pipeline handles its largest messages: each `float32` tensor of 640x640x3 values is 4.9 MB, about four times the size of the 640x640 `rgb8` image, and it crosses three or four nodes before the inference. These tensors are never copied to CPU memory nor serialized. 3. **The decoder** is the first node that copies data to CPU memory: only the output tensors of the network (a few KB for RT-DETR, 2.8 MB for YOLOv8), to filter and decode the detections on the CPU. From this point, the messages are small `Detection2DArray` messages. 4. **The NVIDIA® visualizer** is a Python node that reads the ZED image in CPU memory. The ZED node automatically sends it a CPU copy of the image, while the Resize node keeps receiving the GPU memory. This copy has a cost for each image: disable the visualizer with `visualizer:=false` when you do not need it. The other CPU subscribers of the ZED topics, such as RViz2, `ros2 bag`, and the Foxglove bridge, are served the same way as the visualizer, without affecting the zero-copy path of the Isaac™ ROS nodes. > **Note** > > Without the `cuda_buffer_backend` plugin, the example still works: the ZED node publishes the images in CPU memory, and the Resize node uploads them to the GPU. The ZED node reports the transport in use at startup, for example `* Buffer transport: CUDA zero-copy`. ## Requirements * An Isaac™ ROS 5 environment with ZED support, in Docker or in Bare Metal mode, set up by following the [Setting up Isaac™ ROS 5](/docs/integrations/isaac-ros/setup-and-installation/) and [Use ZED with Isaac™ ROS](/docs/integrations/isaac-ros/use-zed-with-isaac-ros/) guides. * The `rmw_fastrtps_cpp` (Fast DDS, default) or the `rmw_zenoh_cpp` v0.10.5 or newer (Zenoh) ROS 2 middleware, as described in [Middleware requirement](/docs/integrations/isaac-ros/setup-and-installation/#middleware-requirement). * The **ZED ROS 2 Wrapper v5.6.0** or newer, with the CUDA zero-copy transport set up as described in [Set up the CUDA zero-copy transport](/docs/integrations/isaac-ros/cuda-zero-copy/#set-up-the-cuda-zero-copy-transport): the `cuda_buffer` package installed, the `cuda_buffer_backend` plugin built from source, and the wrapper rebuilt. > **Warning** > > Use Fast DDS to run this example. With Zenoh (`rmw_zenoh_cpp` v0.10.5), the detections can stop after a few seconds or a few minutes, without any error: the ZED node keeps publishing the images, but the detection nodes no longer receive them. The issue happens with both the CUDA zero-copy and the CPU transports, and it is under investigation. The [CUDA zero-copy examples](/docs/integrations/isaac-ros/cuda-zero-copy/) are not affected. ## Set up the example Run all the commands in the Isaac™ ROS environment. With the Docker installation, run `isaac-ros activate` to open a shell in the running container: the workspace `${ISAAC_ROS_WS}` of the host is mounted in the container, so the source code, the build artifacts, and the models persist across container restarts. ### Get the source code If the ZED ROS 2 Examples are not in your workspace yet, clone them in the `src` folder, next to the ZED ROS 2 Wrapper: ```bash cd ${ISAAC_ROS_WS}/src git clone https://github.com/stereolabs/zed-ros2-examples.git ``` ### Install the dependencies The example uses the Isaac™ ROS image processing, tensor processing, TensorRT, RT-DETR, and YOLOv8 packages, `vision_msgs`, `foxglove_msgs`, the tool to download the RT-DETR model, and the optional Foxglove bridge. #### Docker The packages installed with `apt-get` in the container are lost when the container is removed, for example after rebuilding the image with `isaac-ros activate --build-local`. Add them to a custom image layer, as you did for the ZED SDK: 1. On the host, create the `${ISAAC_ROS_WS}/docker/Dockerfile.01-zed_object_detection` file: ```dockerfile ARG BASE_IMAGE FROM ${BASE_IMAGE} RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ apt-get update && apt-get install -y --no-install-recommends \ ros-${ROS_DISTRO}-isaac-ros-image-proc \ ros-${ROS_DISTRO}-isaac-ros-tensor-proc \ ros-${ROS_DISTRO}-isaac-ros-tensor-rt \ ros-${ROS_DISTRO}-isaac-ros-rtdetr \ ros-${ROS_DISTRO}-isaac-ros-rtdetr-models-install \ ros-${ROS_DISTRO}-isaac-ros-yolov8 \ ros-${ROS_DISTRO}-vision-msgs \ ros-${ROS_DISTRO}-foxglove-msgs \ ros-${ROS_DISTRO}-foxglove-bridge \ && rm -rf /var/lib/apt/lists/* ``` 2. Add the `01-zed_object_detection` image key to the `${ISAAC_ROS_WS}/.isaac-ros-cli/config.yaml` file: ```yaml docker: image: additional_image_keys: - 00-zed_sdk - 01-zed_object_detection ``` > **Note** > > The Isaac™ ROS CLI builds the custom layers in the alphabetical order of their image keys: the `01-` prefix builds the object detection layer on top of the `00-zed_sdk` layer, which is not built again. See the [note about the build order](/docs/integrations/isaac-ros/use-zed-with-isaac-ros/#add-the-zed-sdk-to-the-isaac-ros-docker-image). The Isaac™ ROS TensorRT packages install the full CUDA toolkit: on Jetson™, the image grows from about 11 GB to about 30 GB. 3. Exit all the shells of the running container, then rebuild the image and start the container: ```bash isaac-ros activate --build-local ``` #### Bare Metal ```bash cd ${ISAAC_ROS_WS} sudo apt-get update rosdep update rosdep install --from-paths src/zed-ros2-wrapper \ src/zed-ros2-examples/isaac_ros/isaac_ros_5/zed_isaac_ros_object_detection \ --ignore-src -r -y sudo apt-get install -y ros-lyrical-isaac-ros-rtdetr-models-install ros-lyrical-foxglove-bridge ``` The ZED ROS 2 Wrapper folder is listed so that `rosdep` uses the wrapper built in your workspace, instead of looking for its Debian packages. ### Install the models The models are stored in the `${ISAAC_ROS_WS}/isaac_ros_assets/models` folder, which persists across container restarts with the Docker installation. The TensorRT engine file (`.plan`) is optimized for the GPU and the TensorRT version of the device where it is generated: generate it on the device that runs the example, and generate it again after a JetPack or TensorRT upgrade. #### RT-DETR Download the SyntheticaDETR `sdetr_grasp` model from NVIDIA® NGC and generate its TensorRT engine. The script asks you to accept the model license (EULA): ```bash ros2 run isaac_ros_rtdetr_models_install install_rtdetr_models.sh --eula ``` The model is saved in the `${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/` folder. The engine generation takes about 2 minutes on a Jetson™ AGX Orin. #### YOLOv8 (optional) Export a YOLOv8 model to ONNX with the Ultralytics Python package. The following commands export the `yolov8s` model, trained on the COCO dataset, in a dedicated Python virtual environment: ```bash mkdir -p ${ISAAC_ROS_WS}/isaac_ros_assets/models/yolov8 cd ${ISAAC_ROS_WS}/isaac_ros_assets/models/yolov8 sudo apt-get install -y python3-venv libgl1 python3 -m venv /tmp/yolov8_export /tmp/yolov8_export/bin/pip install "numpy<2" ultralytics onnx onnxslim /tmp/yolov8_export/bin/python -c "from ultralytics import YOLO; YOLO('yolov8s.pt').export(format='onnx')" rm -rf /tmp/yolov8_export ``` The virtual environment keeps the Ultralytics packages and their dependencies, such as PyTorch and OpenCV, out of the system Python. This is important with the Bare Metal installation, where they could replace the versions required by Isaac™ ROS. The TensorRT engine `yolov8s.plan` is generated automatically from `yolov8s.onnx` the first time you start the example with `model:=yolov8`, which takes about 2 minutes on a Jetson™ AGX Orin. You can also generate it in advance: ```bash /usr/src/tensorrt/bin/trtexec --onnx=yolov8s.onnx --saveEngine=yolov8s.plan ``` > **Note** > > The Ultralytics YOLOv8 models are released under the [AGPL-3.0 license](https://www.ultralytics.com/license). Check that it is compatible with your application, or use the RT-DETR model. ### Build the example ```bash cd ${ISAAC_ROS_WS} colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release \ --packages-up-to zed_isaac_ros_object_detection source install/local_setup.bash ``` Run `source ${ISAAC_ROS_WS}/install/local_setup.bash` in every new shell. With the Docker installation, run `isaac-ros activate` first in every new terminal. ## Run the example Start the example with the RT-DETR model, replacing `` with the model of your camera, for example `zedxonegs`, `zedxone4k`, `zedxm`, or `zed2i`: ```bash ros2 launch zed_isaac_ros_object_detection zed_isaac_ros_object_detection.launch.py \ camera_model:= ``` Start the example with the YOLOv8 model: ```bash ros2 launch zed_isaac_ros_object_detection zed_isaac_ros_object_detection.launch.py \ camera_model:= model:=yolov8 ``` The detections are published when the TensorRT node logs `TensorRT Node initialized successfully`. > **Tip** > > The `sdetr_grasp` RT-DETR model only detects the [objects listed in its model card](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/models/synthetica_detr), and its default confidence threshold is high (`0.9`). If no object is detected, place one of these objects in front of the camera, or lower the threshold, for example with `confidence_threshold:=0.6`. To test the example with common objects, such as people, use the YOLOv8 model. > **Note** > > When you stop the example with `Ctrl+C`, the object detection container can end with `process has died [...] exit code -11`, or `ros2 launch` can stop it with `SIGKILL` after `failed to terminate ... escalating`. This is a [known shutdown issue](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common/issues/246) of the Isaac™ ROS 5.0 nodes, which crash while the container unloads them, and it has no effect on the detections. The NVIDIA® visualizer can also print a `Traceback` when it stops. ### Launch arguments | Argument | Default | Description | | ---------------------- | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `camera_model` | (required) | The model of the camera. | | `model` | `rtdetr` | The object detection model: `rtdetr` or `yolov8`. | | `model_file_path` | see description | The absolute path to the ONNX model file. Default: `${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.onnx` for `rtdetr`, `${ISAAC_ROS_WS}/isaac_ros_assets/models/yolov8/yolov8s.onnx` for `yolov8`. | | `engine_file_path` | see description | The absolute path to the TensorRT engine file, generated from the ONNX model file if it does not exist. Default: the path of the ONNX model file with the `.plan` extension. | | `force_engine_update` | `false` | Regenerate the TensorRT engine file, even if it exists. | | `confidence_threshold` | `0.9` (`rtdetr`), `0.25` (`yolov8`) | The minimum confidence of the published detections. | | `nms_threshold` | `0.45` | YOLOv8 only: the IoU threshold of the Non-Maximum Suppression. | | `num_classes` | `80` | YOLOv8 only: the number of classes of the model. | | `visualizer` | `true` | Start the NVIDIA® visualizer. | | `foxglove` | `false` | Start the Foxglove bridge. | | `foxglove_port` | `8765` | The port of the Foxglove bridge. | | `svo_path` | `live` | Path to an SVO file, to run the example without a camera. | To use your own YOLOv8 model, set `model_file_path` and `engine_file_path`, and `num_classes` if the model is not trained on the COCO dataset. The visualizers display the COCO class names. ### Topics All the topics of the example are in the `/zed_isaac/object_detection` namespace: | Topic | Type | Description | | ------------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `detections` | `vision_msgs/Detection2DArray` | The detected objects, in the coordinates of the ZED image `/zed_isaac/zed/rgb/color/rect/image`. Use this topic in your application. | | `detections_output` | `vision_msgs/Detection2DArray` | The detected objects as published by the Isaac™ ROS decoder, in the coordinates of the 640x640 network image `padded_image`. | | `annotations` | `foxglove_msgs/ImageAnnotations` | The bounding boxes and the labels, to display over the ZED image in Foxglove. | | `processed_image` | `sensor_msgs/Image` | The ZED image with the detections, drawn by the NVIDIA® visualizer. | | `padded_image` | `sensor_msgs/Image` | The ZED image resized and padded to the network input size (640x640). | The class ID of each detection is in `results[0].hypothesis.class_id`, and its confidence in `results[0].hypothesis.score`. ## Visualize the detections ### NVIDIA® visualizer With `visualizer:=true` (default), the NVIDIA® visualizer of the selected model draws the detections over the ZED image and publishes the result on the `/zed_isaac/object_detection/processed_image` topic. Display it with `rqt_image_view`: ```bash sudo apt-get install -y ros-lyrical-rqt-image-view ros2 run rqt_image_view rqt_image_view /zed_isaac/object_detection/processed_image ``` #### RT-DETR ![The detections of the RT-DETR model drawn by the NVIDIA visualizer, displayed in rqt\_image\_view](/_fern-files/_img/15420c618d79697376a0df64465373ef2a1e15a5a3364fc00765f1080185f4f6.webp) *Figure 2: the RT-DETR detections drawn by the NVIDIA® visualizer, captured with a ZED X One S equipped with a fisheye lens. The labels show the class IDs of the `sdetr_grasp` model.* #### YOLOv8 ![The detections of the YOLOv8 model drawn by the NVIDIA visualizer, displayed in rqt\_image\_view](/_fern-files/_img/2240bca9195e7081448443b6782bc0839979cec73ea3d2c6e713ba04c00d418d.webp) *Figure 3: the YOLOv8 detections drawn by the NVIDIA® visualizer, captured with a ZED X One S equipped with a fisheye lens. The labels show the COCO class names and the confidences.* The visualizer processes the images in CPU memory, as shown in [Figure 1](#the-detection-pipeline): set `visualizer:=false` to save CPU resources when you do not need it. ### Foxglove Start the example with the Foxglove bridge: ```bash ros2 launch zed_isaac_ros_object_detection zed_isaac_ros_object_detection.launch.py \ camera_model:= foxglove:=true ``` Then, in [Foxglove](/docs/integrations/foxglove/): 1. Open a connection to `ws://:8765`, where `` is the IP address of the device running the example. The Isaac™ ROS container uses the network of the host, so the address is the same with the Docker installation. 2. Import the layout file `foxglove/zed_isaac_ros_object_detection.json` of the example, from the layouts menu, with the **Import from file...** button. The layout displays the compressed ZED image with the bounding boxes and the labels of the `annotations` topic, the 640x640 network image, and the content of the `detections` topic. Foxglove draws the annotations, so the images are not processed in the ROS 2 graph. #### RT-DETR ![The RT-DETR detections displayed in Foxglove with the layout of the example](/_fern-files/_img/bade641574921d1d568431752f1dc0ab7b6c3bee4fb6e92bb88a206d879e925a.webp) *Figure 4: the RT-DETR detections in Foxglove, captured with a ZED X One S equipped with a fisheye lens. Left: the ZED image with the annotations, labeled with the class IDs and the confidences. Right: the padded 640x640 network image and the `detections` topic.* #### YOLOv8 ![The YOLOv8 detections displayed in Foxglove with the layout of the example](/_fern-files/_img/b1958e4c267e963ea6293dfb08061a354258449cc0f5efcafd6f2b299bfaa040.webp) *Figure 5: the YOLOv8 detections in Foxglove, captured with a ZED X One S equipped with a fisheye lens. Left: the ZED image with the annotations, labeled with the COCO class names and the confidences. Right: the padded 640x640 network image and the `detections` topic.* ## The launch file explained The launch file is located in the folder `launch` and is named `zed_isaac_ros_object_detection.launch.py`. These are the parts that matter for the CUDA zero-copy transport. ### ZED camera parameters The file `config/zed_params.yaml` overrides the parameters of the ZED node: ```yaml /**: ros__parameters: general: pub_resolution: 'CUSTOM' pub_downscale_factor: 2.0 pub_frame_rate: 30.0 buffer_backend: 'cuda' video: enable_24bit_output: true depth: depth_mode: 'NONE' ``` * `buffer_backend: 'cuda'` publishes the images with the CUDA zero-copy transport. * `enable_24bit_output: true` publishes `bgr8` images, the format expected by the Isaac™ ROS image processing nodes, instead of `bgra8`. * `pub_downscale_factor: 2.0` halves the size of the published images: they are resized to 640x640 before the inference anyway. * `depth_mode: 'NONE'` disables the depth processing of the stereo cameras, not used by the example, to save GPU resources. ### Two component containers The ZED node runs in the `/zed_isaac/zed_container` component container, and the Isaac™ ROS nodes in the `/zed_isaac/object_detection/object_detection_container` component container: **`Python`** ```python Python container_name = 'zed_container' od_container_name = 'object_detection_container' for name, ns in [(container_name, namespace_val), (od_container_name, od_namespace)]: actions.append(ComposableNodeContainer( name=name, namespace=ns, package='rclcpp_components', executable='component_container', arguments=['--executor-type', 'multi-threaded', '--ros-args', '--log-level', 'info', '--log-level', 'serialize_buffer_with_endpoint:=warn', '--log-level', 'deserialize_buffer_with_endpoint:=warn'], output='screen', )) ``` * In the **object detection container**, the Isaac™ ROS nodes exchange their CUDA buffers with rclcpp Intra-Process Communication, enabled by each Isaac™ ROS node on its publishers and subscriptions. * Between the two containers, the `cuda` buffer backend shares the ZED images through CUDA IPC (inter-process communication), still without copies. The `serialize_buffer_with_endpoint` and `deserialize_buffer_with_endpoint` loggers of the ROS 2 buffer typesupport print several `INFO` lines for each CUDA buffer message, thousands of lines per minute. The containers raise their level to `WARN` to keep the log readable. The Isaac™ ROS `ResizeNode`, which receives the ZED images, is configured with `input_qos_depth: 1` (subscription queue) and `input_queue_size: 1` (image and camera info synchronization queue), so that it always takes the latest image. The `cuda` buffer backend can reuse the GPU memory of an image about 100 ms after it is published, so an image taken late from a longer queue could no longer be read. The same applies to every Isaac™ ROS node that receives CUDA buffers. See the [warning about the queue depth](/docs/integrations/isaac-ros/cuda-zero-copy/images/#accept-the-cuda-buffer-backend) in the CUDA zero-copy tutorial. > **Warning** > > Keep the ZED node and the Isaac™ ROS nodes in two different containers. With `cuda_buffer_backend` 0.1.2, the same-process path can deliver an image buffer that the ZED node has already reused for a new image: the backend drops it, the Isaac™ ROS node receives an empty buffer, and the container stops. The queue depth of `1` does not prevent this error when the nodes share the ZED container. We tested both layouts on a Jetson™ AGX Orin, alternating the two models, with the Docker and the Bare Metal installations: | Layout | `ResizeNode` `input_queue_size` | Runs | Runs stopped by the error | | ---------------- | ------------------------------- | --------------------------- | ------------------------- | | Single container | `10` | 6 (Docker) | 3 | | Single container | `1` | 16 (8 Docker, 8 Bare Metal) | 9 | | Two containers | `10` | 10 (8 Docker, 2 Bare Metal) | 0 | | Two containers | `1` (the example configuration) | 8 (4 Docker, 4 Bare Metal) | 0 | With two containers, the detections are published at 30 Hz with both models. The ZED node is started with IPC disabled: **`Python`** ```python Python zed_wrapper_launch = IncludeLaunchDescription( launch_description_source=PythonLaunchDescriptionSource([ get_package_share_directory('zed_wrapper'), '/launch/zed_camera.launch.py' ]), launch_arguments={ 'camera_model': camera_model, 'camera_name': camera_name_val, 'container_name': container_name, 'namespace': namespace_val, 'svo_path': svo_path, 'enable_ipc': 'false', 'ros_params_override_path': camera_config_override_path }.items() ) ``` rclcpp Intra-Process Communication delivers the messages without the buffer backend negotiation, so the ZED node publishes CPU data when an intra-process subscriber is connected. With IPC disabled, the ZED node always publishes CUDA buffers to the subscribers that accept them. ### Image encoding The Resize node subscribes directly to the ZED rectified image, and the Pad node to the output of the Resize node. The image is resized keeping its aspect ratio, then padded on the bottom and on the right side, as in the NVIDIA® RT-DETR pipeline: **`Python`** ```python Python resize_node = ComposableNode( package='isaac_ros_image_proc', plugin='nvidia::isaac_ros::image_proc::ResizeNode', name='resize_node', namespace=od_namespace, parameters=[{ 'output_width': NETWORK_IMAGE_SIZE, 'output_height': NETWORK_IMAGE_SIZE, 'keep_aspect_ratio': True, 'disable_padding': True }], remappings=[ ('image', zed_image_topic), ('camera_info', zed_camera_info_topic) ] ) pad_node = ComposableNode( package='isaac_ros_image_proc', plugin='nvidia::isaac_ros::image_proc::PadNode', name='pad_node', namespace=od_namespace, parameters=[{ 'output_image_width': NETWORK_IMAGE_SIZE, 'output_image_height': NETWORK_IMAGE_SIZE, 'padding_type': 'BOTTOM_RIGHT' }], remappings=[ ('image', 'resize/image') ] ) ``` Because the padding is only on the bottom and on the right side, the detections in the network image are mapped back to the ZED image with a single scale factor for each axis. The `zed_detection_rescaler` Python node computes it from the size of the ZED image in the `camera_info` topic, with the same computation as the Resize node. ### Model selection The `model` argument selects the configuration of the TensorRT node and the model-specific nodes: * **RT-DETR**: the Reshape node publishes the tensor to the RT-DETR preprocessor, which adds the size of the input image as a second tensor, then the RT-DETR decoder converts the output tensors into detections. The image tensor keeps the pixel values in \[0, 255]. * **YOLOv8**: the Reshape node publishes the tensor directly to the TensorRT node, and the YOLOv8 decoder applies the Non-Maximum Suppression. The image tensor is scaled to \[0, 1]. ## Troubleshooting * **No detection is published**: check that all the terminals use the same supported middleware, as described in [Middleware requirement](/docs/integrations/isaac-ros/setup-and-installation/#middleware-requirement). With the RT-DETR model, also read the tip in [Run the example](#run-the-example). * **The example stops with `Neither the TensorRT engine file ... nor the ONNX model file ... exist`**: install the model as described in [Install the models](#install-the-models), or set the `model_file_path` and `engine_file_path` launch arguments. * **The example stops with `The ISAAC_ROS_WS environment variable is not set`**: set `ISAAC_ROS_WS` to the path of your Isaac™ ROS workspace, as described in [Create the Isaac™ ROS workspace](/docs/integrations/isaac-ros/setup-and-installation/#create-the-isaac-ros-workspace). * **The ZED node opens another camera or another SVO file, or fails with `INVALID SVO FILE` while the file is valid**: another device on the same network runs the example. The component containers of the two devices have the same names, so `ros2 launch` can load the nodes in the containers of the other device. Set a different `ROS_DOMAIN_ID` on each device, for example `export ROS_DOMAIN_ID=11`, in all the terminals of the device. * **TensorRT reports an error while loading the engine file**: the engine was generated on another device or with another TensorRT version. Regenerate it with `force_engine_update:=true`. * **The NVIDIA® visualizer stops with `ImportError: numpy.core.multiarray failed to import`**: a NumPy 2 version installed with `pip` in your user folder (`~/.local/lib/python3.12/site-packages`) replaces the NumPy 1 version used by `cv_bridge`. Check it with `python3 -c "import numpy; print(numpy.__version__, numpy.__file__)"`, then remove the user version with `pip uninstall numpy`, or start the example with the `PYTHONNOUSERSITE=1` environment variable to ignore the user Python packages. ## Conclusion This tutorial showed how to detect objects in the ZED images with the Isaac™ ROS 5 RT-DETR and YOLOv8 packages. With the CUDA zero-copy transport, the image goes from the ZED SDK to the TensorRT inference without leaving the GPU memory: the ZED node shares it with the Isaac™ ROS nodes as a small descriptor, and the Isaac™ ROS nodes pass their image and tensor buffers by pointer. Only the inference results, and the images requested by CPU subscribers, are copied to CPU memory. To process the ZED data in GPU memory in your own nodes, follow the [CUDA zero-copy image tutorial](/docs/integrations/isaac-ros/cuda-zero-copy/images/).