Skip to content

Commit

Permalink
Merge pull request #32 from NVIDIA-ISAAC-ROS/release-3.2
Browse files Browse the repository at this point in the history
Isaac ROS 3.2
  • Loading branch information
jaiveersinghNV authored Dec 11, 2024
2 parents 6058ca5 + 9e0f216 commit 7cdfd6e
Show file tree
Hide file tree
Showing 33 changed files with 672 additions and 82 deletions.
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,11 +44,11 @@ This package is powered by [NVIDIA Isaac Transport for ROS (NITROS)](https://dev

## Performance

| Sample Graph<br/><br/> | Input Size<br/><br/> | AGX Orin<br/><br/> | Orin NX<br/><br/> | Orin Nano 8GB<br/><br/> | x86_64 w/ RTX 4060 Ti<br/><br/> | x86_64 w/ RTX 4090<br/><br/> |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [SAM Image Segmentation Graph](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/benchmarks/isaac_ros_segment_anything_benchmark/scripts/isaac_ros_segment_anything_graph.py)<br/><br/><br/>Full SAM<br/><br/> | 720p<br/><br/><br/><br/> | [2.22 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_sam_graph-agx_orin.json)<br/><br/><br/>470 ms @ 30Hz<br/><br/> | –<br/><br/><br/><br/> | –<br/><br/><br/><br/> | –<br/><br/><br/><br/> | [14.6 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_sam_graph-x86_4090.json)<br/><br/><br/>79 ms @ 30Hz<br/><br/> |
| [SAM Image Segmentation Graph](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/benchmarks/isaac_ros_segment_anything_benchmark/scripts/isaac_ros_mobile_segment_anything_graph.py)<br/><br/><br/>Mobile SAM<br/><br/> | 720p<br/><br/><br/><br/> | [10.8 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_mobile_sam_graph-agx_orin.json)<br/><br/><br/>880 ms @ 30Hz<br/><br/> | [5.13 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_mobile_sam_graph-orin_nx.json)<br/><br/><br/>1500 ms @ 30Hz<br/><br/> | [2.22 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_mobile_sam_graph-orin_nano.json)<br/><br/><br/>360 ms @ 30Hz<br/><br/> | [27.0 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_mobile_sam_graph-nuc_4060ti.json)<br/><br/><br/>62 ms @ 30Hz<br/><br/> | [60.3 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_mobile_sam_graph-x86_4090.json)<br/><br/><br/>27 ms @ 30Hz<br/><br/> |
| [TensorRT Graph](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/benchmarks/isaac_ros_unet_benchmark/scripts/isaac_ros_unet_graph.py)<br/><br/><br/>PeopleSemSegNet<br/><br/> | 544p<br/><br/><br/><br/> | [371 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_unet_graph-agx_orin.json)<br/><br/><br/>19 ms @ 30Hz<br/><br/> | [250 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_unet_graph-orin_nx.json)<br/><br/><br/>20 ms @ 30Hz<br/><br/> | [163 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_unet_graph-orin_nano.json)<br/><br/><br/>23 ms @ 30Hz<br/><br/> | [670 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_unet_graph-nuc_4060ti.json)<br/><br/><br/>11 ms @ 30Hz<br/><br/> | [688 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_unet_graph-x86_4090.json)<br/><br/><br/>9.3 ms @ 30Hz<br/><br/> |
| Sample Graph<br/><br/> | Input Size<br/><br/> | AGX Orin<br/><br/> | Orin NX<br/><br/> | Orin Nano 8GB<br/><br/> | x86_64 w/ RTX 4090<br/><br/> |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [SAM Image Segmentation Graph](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/benchmarks/isaac_ros_segment_anything_benchmark/scripts/isaac_ros_segment_anything_graph.py)<br/><br/><br/>Full SAM<br/><br/> | 720p<br/><br/><br/><br/> | [2.22 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_sam_graph-agx_orin.json)<br/><br/><br/>390 ms @ 30Hz<br/><br/> | –<br/><br/><br/><br/> | –<br/><br/><br/><br/> | [16.4 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_sam_graph-x86-4090.json)<br/><br/><br/>280 ms @ 30Hz<br/><br/> |
| [SAM Image Segmentation Graph](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/benchmarks/isaac_ros_segment_anything_benchmark/scripts/isaac_ros_mobile_segment_anything_graph.py)<br/><br/><br/>Mobile SAM<br/><br/> | 720p<br/><br/><br/><br/> | [8.75 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_mobile_sam_graph-agx_orin.json)<br/><br/><br/>570 ms @ 30Hz<br/><br/> | [5.34 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_mobile_sam_graph-orin_nx.json)<br/><br/><br/>1400 ms @ 30Hz<br/><br/> | [2.22 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_mobile_sam_graph-orin_nano.json)<br/><br/><br/>340 ms @ 30Hz<br/><br/> | [68.6 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_mobile_sam_graph-x86-4090.json)<br/><br/><br/>23 ms @ 30Hz<br/><br/> |
| [TensorRT Graph](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/benchmarks/isaac_ros_unet_benchmark/scripts/isaac_ros_unet_graph.py)<br/><br/><br/>PeopleSemSegNet<br/><br/> | 544p<br/><br/><br/><br/> | [371 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_unet_graph-agx_orin.json)<br/><br/><br/>19 ms @ 30Hz<br/><br/> | [250 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_unet_graph-orin_nx.json)<br/><br/><br/>20 ms @ 30Hz<br/><br/> | [163 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_unet_graph-orin_nano.json)<br/><br/><br/>23 ms @ 30Hz<br/><br/> | [670 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_unet_graph-nuc_4060ti.json)<br/><br/><br/>11 ms @ 30Hz<br/><br/> |

---

Expand Down Expand Up @@ -78,4 +78,4 @@ Please visit the [Isaac ROS Documentation](https://nvidia-isaac-ros.github.io/re

## Latest

Update 2024-09-26: Update for ZED compatibility
Update 2024-12-10: Update to be compatible with JetPack 6.1
Original file line number Diff line number Diff line change
Expand Up @@ -63,4 +63,10 @@ set_target_properties(${PROJECT_NAME} PROPERTIES
# Install the binary file
install(TARGETS ${PROJECT_NAME} DESTINATION share/${PROJECT_NAME}/gxf/lib)


# Embed versioning information into installed files
ament_index_get_resource(ISAAC_ROS_COMMON_CMAKE_PATH isaac_ros_common_cmake_path isaac_ros_common)
include("${ISAAC_ROS_COMMON_CMAKE_PATH}/isaac_ros_common-version-info.cmake")
generate_version_info(${PROJECT_NAME})

ament_auto_package(INSTALL_TO_SHARE)
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ SPDX-License-Identifier: Apache-2.0
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="3">
<name>gxf_isaac_ros_segment_anything</name>
<version>3.1.0</version>
<version>3.2.0</version>
<description>Segmentation Anything extension.</description>

<maintainer email="[email protected]">Isaac ROS Maintainers</maintainer>
Expand Down
6 changes: 6 additions & 0 deletions isaac_ros_gxf_extensions/gxf_isaac_ros_unet/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -70,5 +70,11 @@ set_target_properties(${PROJECT_NAME} PROPERTIES
# Install the binary file
install(TARGETS ${PROJECT_NAME} DESTINATION share/${PROJECT_NAME}/gxf/lib)


# Embed versioning information into installed files
ament_index_get_resource(ISAAC_ROS_COMMON_CMAKE_PATH isaac_ros_common_cmake_path isaac_ros_common)
include("${ISAAC_ROS_COMMON_CMAKE_PATH}/isaac_ros_common-version-info.cmake")
generate_version_info(${PROJECT_NAME})

ament_auto_package(INSTALL_TO_SHARE)

2 changes: 1 addition & 1 deletion isaac_ros_gxf_extensions/gxf_isaac_ros_unet/package.xml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ SPDX-License-Identifier: Apache-2.0
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="3">
<name>gxf_isaac_ros_unet</name>
<version>3.1.0</version>
<version>3.2.0</version>
<description>Segmentation post-processor extension.</description>

<maintainer email="[email protected]">Isaac ROS Maintainers</maintainer>
Expand Down
6 changes: 6 additions & 0 deletions isaac_ros_peoplesemseg_models_install/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -29,4 +29,10 @@ install_isaac_ros_asset(install_peoplesemsegnet_shuffleseg)
install(PROGRAMS asset_scripts/install_peoplesemsegnet_shuffleseg.sh DESTINATION lib/${PROJECT_NAME})
install(PROGRAMS asset_scripts/install_peoplesemsegnet_vanilla.sh DESTINATION lib/${PROJECT_NAME})


# Embed versioning information into installed files
ament_index_get_resource(ISAAC_ROS_COMMON_CMAKE_PATH isaac_ros_common_cmake_path isaac_ros_common)
include("${ISAAC_ROS_COMMON_CMAKE_PATH}/isaac_ros_common-version-info.cmake")
generate_version_info(${PROJECT_NAME})

ament_auto_package(INSTALL_TO_SHARE)
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,17 @@
# distribution of this software and related documentation without an express
# license agreement from NVIDIA CORPORATION is strictly prohibited.

# Download and tao-convert ESS models.
# Download and TRT-compile PeopleSemSeg-ShuffleSeg models.
# * Models will be stored in the isaac_ros_assets dir
# * The script must be called with the --eula argument prior to downloading.

set -e

ASSET_NAME="deployable_shuffleseg_unet_amr_v1.0"
EULA_URL="https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/peoplesemsegnet_amr"
ASSET_NAME="optimized_deployable_shuffleseg_unet_amr_v1.0"
EULA_URL="https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/models/optimized-peoplesemseg-amr"
ASSET_DIR="${ISAAC_ROS_WS}/isaac_ros_assets/models/peoplesemsegnet/${ASSET_NAME}"
ASSET_INSTALL_PATHS="${ASSET_DIR}/1/model.plan"
MODEL_URL="https://api.ngc.nvidia.com/v2/models/org/nvidia/team/tao/peoplesemsegnet_amr/deployable_v1.1/files?redirect=true&path=peoplesemsegnet_amr_rel.onnx"
MODEL_URL="https://api.ngc.nvidia.com/v2/models/org/nvidia/team/isaac/optimized-peoplesemseg-amr/v1.0/files?redirect=true&path=model.onnx"

source "isaac_ros_asset_eula.sh"

Expand All @@ -27,9 +27,9 @@ wget "${MODEL_URL}" -O "${ASSET_DIR}/model.onnx"

echo "Converting PeopleSemSegnet shuffleseg amr onnx file to plan file."
/usr/src/tensorrt/bin/trtexec \
--maxShapes="input_2:0":1x3x544x960 \
--minShapes="input_2:0":1x3x544x960 \
--optShapes="input_2:0":1x3x544x960 \
--maxShapes="input_2":1x544x960x3 \
--minShapes="input_2":1x544x960x3 \
--optShapes="input_2":1x544x960x3 \
--fp16 \
--saveEngine="${ASSET_INSTALL_PATHS}" \
--onnx="${ASSET_DIR}/model.onnx"
Expand All @@ -41,9 +41,9 @@ platform: "tensorrt_plan"
max_batch_size: 0
input [
{
name: "input_2:0"
name: "input_2"
data_type: TYPE_FP32
dims: [ 1, 3, 544, 960 ]
dims: [ 1, 544, 960, 3 ]
}
]
output [
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,33 +7,33 @@
# distribution of this software and related documentation without an express
# license agreement from NVIDIA CORPORATION is strictly prohibited.

# Download and tao-convert ESS models.
# Download and TRT-compile PeopleSemSeg-Vanilla models.
# * Models will be stored in the isaac_ros_assets dir
# * The script must be called with the --eula argument prior to downloading.

set -e

ASSET_NAME="deployable_quantized_vanilla_unet_v2.0"
ASSET_NAME="deployable_quantized_vanilla_unet_onnx_v2.0"
EULA_URL="https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/peoplesemsegnet"
ASSET_DIR="${ISAAC_ROS_WS}/isaac_ros_assets/models/peoplesemsegnet/${ASSET_NAME}"
ASSET_INSTALL_PATHS="${ASSET_DIR}/1/model.plan"
MODEL_URL="https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplesemsegnet/versions/deployable_quantized_vanilla_unet_v2.0/files/peoplesemsegnet_vanilla_unet_dynamic_etlt_int8_fp16.etlt"
MODEL_URL="https://api.ngc.nvidia.com/v2/models/org/nvidia/team/tao/peoplesemsegnet/deployable_quantized_vanilla_unet_onnx_v2.0/files?redirect=true&path=peoplesemsegnet_vanilla_unet_dynamic_etlt_int8_fp16.onnx"

source "isaac_ros_asset_eula.sh"

mkdir -p $(dirname "$ASSET_INSTALL_PATHS")

wget "${MODEL_URL}" -O "${ASSET_DIR}/model.etlt"
wget "${MODEL_URL}" -O "${ASSET_DIR}/model.onnx"

echo "Converting PeopleSemSegnet etlt file to plan file."
/opt/nvidia/tao/tao-converter \
-k tlt_encode \
-d 3,544,960 \
-p input_1:0,1x3x544x960,1x3x544x960,1x3x544x960 \
-t fp16 \
-e "${ASSET_INSTALL_PATHS}" \
-o argmax_1 \
"${ASSET_DIR}/model.etlt"
echo "Converting PeopleSemSegnet onnx file to plan file."
/usr/src/tensorrt/bin/trtexec \
--maxShapes="input_1:0":1x3x544x960 \
--minShapes="input_1:0":1x3x544x960 \
--optShapes="input_1:0":1x3x544x960 \
--onnx="${ASSET_DIR}/model.onnx" \
--saveEngine="${ASSET_DIR}/1/model.plan" \
--fp16 \
--skipInference

# Create config.pbtxt
config_file_text=$(
Expand Down
2 changes: 1 addition & 1 deletion isaac_ros_peoplesemseg_models_install/package.xml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ SPDX-License-Identifier: Apache-2.0
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="3">
<name>isaac_ros_peoplesemseg_models_install</name>
<version>3.1.0</version>
<version>3.2.0</version>
<description>Scripts for installing people segmentation models</description>

<maintainer email="[email protected]">Isaac ROS Maintainers</maintainer>
Expand Down
6 changes: 6 additions & 0 deletions isaac_ros_segformer/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -40,4 +40,10 @@ if(BUILD_TESTING)
add_launch_test(test/isaac_ros_segformer_pol_test.py TIMEOUT "300")
endif()


# Embed versioning information into installed files
ament_index_get_resource(ISAAC_ROS_COMMON_CMAKE_PATH isaac_ros_common_cmake_path isaac_ros_common)
include("${ISAAC_ROS_COMMON_CMAKE_PATH}/isaac_ros_common-version-info.cmake")
generate_version_info(${PROJECT_NAME})

ament_auto_package(INSTALL_TO_SHARE launch)
2 changes: 1 addition & 1 deletion isaac_ros_segformer/package.xml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ SPDX-License-Identifier: Apache-2.0
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="3">
<name>isaac_ros_segformer</name>
<version>3.1.0</version>
<version>3.2.0</version>
<description>Segformer model processing</description>

<maintainer email="[email protected]">Isaac ROS Maintainers</maintainer>
Expand Down
2 changes: 2 additions & 0 deletions isaac_ros_segformer/test/dummy_model/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
*.plan
*.onnx
Empty file.
3 changes: 0 additions & 3 deletions isaac_ros_segformer/test/dummy_model/model.dummy.onnx

This file was deleted.

17 changes: 13 additions & 4 deletions isaac_ros_segformer/test/isaac_ros_segformer_pol_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,20 +32,18 @@
import time

from ament_index_python.packages import get_package_share_directory
from isaac_ros_test import IsaacROSBaseTest, JSONConversion
from isaac_ros_test import IsaacROSBaseTest, JSONConversion, MockModelGenerator
import launch
from launch.actions import IncludeLaunchDescription
from launch.launch_description_sources import PythonLaunchDescriptionSource
from launch_ros.actions.composable_node_container import ComposableNodeContainer
from launch_ros.descriptions.composable_node import ComposableNode
import launch_testing

import numpy as np

import pytest
import rclpy

from sensor_msgs.msg import CameraInfo, Image
import torch

_TEST_CASE_NAMESPACE = 'segformer_node_test'

Expand Down Expand Up @@ -74,6 +72,17 @@ def generate_test_description():
if e.errno != errno.ENOENT:
print('File exists but error deleting /tmp/trt_engine.plan')

# Generate a mock model with SegFormer-like I/O
MockModelGenerator.generate(
input_bindings=[
MockModelGenerator.Binding('input', [-1, 3, 512, 512], torch.float32)
],
output_bindings=[
MockModelGenerator.Binding('output', [-1, 1, 512, 512], torch.int64),
],
output_onnx_path=model_file_path
)

encoder_dir = get_package_share_directory('isaac_ros_dnn_image_encoder')
encoder_node_launch = IncludeLaunchDescription(
PythonLaunchDescriptionSource(
Expand Down
6 changes: 6 additions & 0 deletions isaac_ros_segment_anything/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -79,4 +79,10 @@ if(BUILD_TESTING)
add_launch_test(test/isaac_ros_segment_anything_point_prompt_pol.py TIMEOUT "300")
endif()


# Embed versioning information into installed files
ament_index_get_resource(ISAAC_ROS_COMMON_CMAKE_PATH isaac_ros_common_cmake_path isaac_ros_common)
include("${ISAAC_ROS_COMMON_CMAKE_PATH}/isaac_ros_common-version-info.cmake")
generate_version_info(${PROJECT_NAME})

ament_auto_package(INSTALL_TO_SHARE config launch)
Loading

0 comments on commit 7cdfd6e

Please sign in to comment.