Tutorial - AprilTag Detection with ZED and Isaac™ ROS

This tutorial demonstrates how to create an Isaac™ ROS application using AprilTag detection as example. It assumes you have already set up the ZED camera and are publishing its data using the zed-ros2-wrapper packages as described in the Setting up Isaac™ ROS to work with ZED Cameras in ROS 2 section.

The application will subscribe to the ZED camera’s image stream, convert the ZED images to a supported format, and finally process the images to detect AprilTags, and publish the detected tags’ information.

The example leverages the NITROS communication framework for efficient data exchange between the ZED camera and the AprilTag detection module through the GPU memory without unnecessary data copies on CPU.

Please refer to the official NVIDIA® documentation for more details on the Image Format Converter and AprilTag Isaac™ ROS nodes.

Setup the example

If you haven’t done so already, set up your development environment by following our NVIDIA® Isaac™ ROS installation guide.

Install the ZED ROS2 Examples packages in the workspace:

$
cd ${ISAAC_ROS_WS}/src && \
>
git clone https://github.com/stereolabs/zed-ros2-examples.git

If you are using Docker, start the Isaac™ ROS environment with ZED support:

$
cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
>
./scripts/run_dev.sh -i ros2_humble.zed \
>
-a "-v /usr/local/zed/settings:/usr/local/zed/settings \
>
    -v /usr/local/zed/resources:/usr/local/zed/resources"

Install the Isaac™ ROS AprilTag and Image Process packages in the workspace:

$
sudo apt-get update && \
>
sudo apt-get install -y ros-humble-isaac-ros-apriltag ros-humble-isaac-ros-image-proc

Build the AprilTag detection application:

$
cd ${ISAAC_ROS_WS} && \
>
colcon build --symlink-install --packages-above zed_isaac_ros_april_tag

Run the AprilTag detection application

Before running the AprilTag detection application you must configure the example.

$
cd ${ISAAC_ROS_WS}/src/zed-ros2-examples/isaac_ros/zed_isaac_ros_april_tag/config

Modify the zed_isaac_ros_april_tag.yaml file to configure the AprilTag detection parameters:

1
/**:
2
    ros__parameters:
3
        size: 0.155
4
        max_tags: 64
5
        tile_size: 4
6
        tag_family: 'tag36h11' # Tag family to detect. CUDA backend only supports tag36h11. CPU and PVA backends support tag36h11, tag16h5, tag25h9, tag36h10, tag36h11, circle21h7, circle49h12, custom48h12, standard41h12, standard52h13
7
        backends: 'CUDA' # Backend to perform detection with. Options include CPU, CUDA, PVA

• size: The size of the AprilTag in meters (e.g., 0.155).
• max_tags: The maximum number of tags to detect (e.g., 64).
• tile_size: The tile/window size for adaptive thresholding in pixels (e.g., 4).
• tag_family: The family of AprilTags to detect. CUDA backend only supports tag36h11. CPU and PVA backends support tag36h11, tag16h5, tag25h9, tag36h10, tag36h11, circle21h7, circle49h12, custom48h12, standard41h12, standard52h13
• backends: The backend to use for detection. Options include CPU, CUDA, PVA.

You can generate the April Tags using the tool at this page.

Modify the zed_params.yaml file to configure the ZED camera parameters:

1
/**:
2
    ros__parameters:
3
        general:
4
          grab_resolution: 'HD1080'
5
          grab_frame_rate: 30
6
          pub_resolution: 'CUSTOM'
7
          pub_downscale_factor: 2.0
8
          pub_frame_rate: 30.0

• grab_resolution: The native camera grab resolution. Options include HD2K, HD1200, HD1080, HD720, SVGA, AUTO.
• grab_frame_rate: The ZED SDK internal grabbing rate. Options include 60, 30, 15 for HD1200/HD1080 and 120, 60, 30, 15 for SVGA.
• pub_resolution: The resolution used for image and depth map publishing. Options include NATIVE and CUSTOM. Set NATIVE to use the same general.grab_resolution; set CUSTOM to apply the general.pub_downscale_factor downscale factory to reduce bandwidth in data transmission.
• pub_downscale_factor: The rescale factor used to rescale the image before publishing when pub_resolution is CUSTOM.
• pub_frame_rate: The data publishing frame rate.

Visualize Results

First, make sure you compiled and built the example as described in the Setup section.

$
cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
>
./scripts/run_dev.sh -i ros2_humble.zed \
>
-a "-v /usr/local/zed/settings:/usr/local/zed/settings \
>
    -v /usr/local/zed/resources:/usr/local/zed/resources"

Start the processing by using the command:

$
ros2 launch zed_isaac_ros_april_tag zed_isaac_ros_april_tag.launch.py camera_model:=<camera_model>

where <camera_model> is the model of the ZED camera you are using (e.g., zedx, zed2i, etc.).

Point the camera to an AprilTag target and observe the AprilTag detection output /tag_detections on a separate terminal with the command:

$
ros2 topic echo /tag_detections

You can visualize the full ROS2 topic graph with:

$
ros2 run rqt_graph rqt_graph

The launch file explained

The launch file launch/zed_isaac_ros_april_tag.launch.py is responsible for starting all the necessary nodes and configurations for the AprilTag detection application. It sets up the parameters, including the ZED camera settings and the AprilTag detection parameters, and launches the required nodes in a ROS 2 component container.

Retrieve the configuration parameters from the relative path config/zed_isaac_ros_april_tag.yaml and config/zed_params.yaml:

1
    # Get the path to the camera configuration file
2
    camera_config_override_path = os.path.join(
3
        get_package_share_directory('zed_isaac_ros_april_tag'),
4
        'config',
5
        'zed_params.yaml'
6
    )
7
8
    # Get the path to the AprilTag configuration file
9
    apriltag_config_path = os.path.join(
10
        get_package_share_directory('zed_isaac_ros_april_tag'),
11
        'config',
12
        'zed_isaac_ros_april_tag.yaml'
13
    )

Create a ROS 2 component container:

1
    # ROS 2 Component Container
2
    container_name = 'zed_container'
3
    info = '* Starting Composable node container: ' + namespace_val + '/' + container_name
4
    actions.append(LogInfo(msg=TextSubstitution(text=info)))
5
6
    # Note: It is crucial that the 'executable' field is set to be 'component_container_mt'
7
    #  so that the created nodes can be started and communicated correctly within the same process.
8
9
    zed_container = ComposableNodeContainer(
10
        name=container_name,
11
        namespace=namespace_val,
12
        package='rclcpp_components',
13
        executable='component_container_mt',
14
        arguments=['--ros-args', '--log-level', 'info'],
15
        output='screen',
16
    )
17
    actions.append(zed_container)

Create a ZED Component and automatically load it into the container_name container previously created:

1
    # ZED Wrapper launch file
2
    zed_wrapper_launch = IncludeLaunchDescription(
3
        launch_description_source=PythonLaunchDescriptionSource([
4
            get_package_share_directory('zed_wrapper'),
5
            '/launch/zed_camera.launch.py'
6
        ]),
7
        launch_arguments={
8
            'camera_model': camera_model,
9
            'container_name': container_name,
10
            'namespace': namespace_val,
11
            'enable_ipc': 'false',
12
            'ros_params_override_path': camera_config_override_path
13
        }.items()
14
    )
15
    actions.append(zed_wrapper_launch)

Setup the Image Format converter resolution by retrieving the information from the camera configuration:

1
    # Read the resolution from the ZED parameters file
2
    with open(camera_config_override_path, 'r') as f:
3
        configuration = yaml.safe_load(f)
4
        print(f'Loaded configuration: {configuration}')
5
        resolution = configuration["/**"]["ros__parameters"]["general"]["grab_resolution"]
6
        pub_resolution = configuration["/**"]["ros__parameters"]["general"]["pub_resolution"]
7
        pub_downscale_factor = configuration["/**"]["ros__parameters"]["general"]["pub_downscale_factor"]
8
        if pub_resolution == 'CUSTOM':
9
            rescale = pub_downscale_factor
10
        else:
11
            rescale = 1.0
12
13
    if resolution == 'HD2K':
14
        image_width = 2208
15
        image_height = 1242
16
    elif resolution == 'HD1200':
17
        image_width = 1280
18
        image_height = 720
19
    elif resolution == 'HD1080':
20
        image_width = 1920
21
        image_height = 1080
22
    elif resolution == 'HD720':
23
        image_width = 1280
24
        image_height = 720
25
    elif resolution == 'SVGA':
26
        image_width = 960
27
        image_height = 600
28
    elif resolution == 'VGA':
29
        image_width = 672
30
        image_height = 376

Create the Image Format Converter component and set the resolution parameter and the required topic name remappings:

1
    # Isaac ROS Node to convert from ZED BGRA8 image to BGR8 required by AprilTag
2
    isaac_converter_node = ComposableNode(
3
        package='isaac_ros_image_proc',
4
        plugin='nvidia::isaac_ros::image_proc::ImageFormatConverterNode',
5
        name='zed_image_converter',
6
        namespace=namespace_val,
7
        parameters=[
8
            {
9
                'image_width': int(image_width / rescale),
10
                'image_height': int(image_height / rescale),
11
                'encoding_desired': 'bgr8',
12
                'num_blocks': 40
13
            }
14
        ],
15
        remappings=[
16
            ('image_raw', 'zed/rgb/color/rect/image'),
17
            ('image', 'zed/rgb/color/rect/image_bgr8')
18
        ]
19
    )

Create the AprilTag component with the required topic name remappings:

1
    # AprilTag detection node
2
    isac_apriltag_node = ComposableNode(
3
        package='isaac_ros_apriltag',
4
        plugin='nvidia::isaac_ros::apriltag::AprilTagNode',
5
        name='apriltag',
6
        namespace=namespace_val,
7
        remappings=[
8
                ('image', 'zed/rgb/color/rect/image_bgr8'),
9
                ('camera_info', 'zed/rgb/camera_info')
10
        ],
11
        parameters=[apriltag_config_path]
12
    )

Load the Image Converter and AprilTag components into the container:

1
    container_full_name = namespace_val + '/' + container_name
2
    # Load the Converter node into the container
3
    load_converter_node = LoadComposableNodes(
4
        composable_node_descriptions=[isaac_converter_node],
5
        target_container=container_full_name
6
    )
7
    actions.append(load_converter_node)
8
9
    # Load the AprilTag node into the container
10
    load_april_tag_node = LoadComposableNodes(
11
        composable_node_descriptions=[isac_apriltag_node],
12
        target_container=container_full_name
13
    )
14
    actions.append(load_april_tag_node)

The launch file accepts two arguments:

1. camera_model: The model of the camera being used (e.g., ZED 2i, ZED 2, etc.).
2. disable_tf: If True disable TF broadcasting for all the cameras in order to fuse visual odometry information externally.

These arguments can be passed when launching the file to customize the behavior of the nodes.

Fix the TF issue (optional)

If you notice that the TF of the detected AprilTags is not correctly propagated, you must download the patched version of the isaac_ros_apriltag package and build it from the source.

Clone the repository into your workspace:

$
cd ${ISAAC_ROS_WS}/src
$
git clone git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_apriltag.git

Fetch the specific Pull Request that contains the fix:

$
cd isaac_ros_apriltag
$
git fetch origin pull/64/head:fix_tf
$
git checkout fix_tf

Build the package:

$
cd ${ISAAC_ROS_WS}
$
colcon build --symlink-install --packages-up-to isaac_ros_apriltag

Now you can run the launch file again, and the TF of the detected AprilTags should be correctly propagated.