rosidl_generate_dds_interfaces() doesn't support passing msg files as of Dashing

[TSC21]

Bug report

Required Info:

• Operating System:
  • Ubuntu 18.04
• Installation type:
  • binaries
• DDS implementation:
  • Fast-RTPS
• ROS 2 release:
  • Dashing
• rosidl-generator-dds-idl version:
  • 0.7.1-1bionic.20191015.013612

Steps to reproduce issue

1. git clone https://github.com/PX4/px4_msgs.git -b use_rosidl_generator_dds_idl to a colcon/ament ws;
2. colcon build --event-handlers console_direct+.

Expected behavior

rosidl_generate_dds_interfaces() should correctly generate the IDL files for each message in px4_msgs, as it was doing for the Crystal release.

Actual behavior

The generator fails with the following error:

[ 13%] Built target px4_msgs__rosidl_generator_c
[ 13%] Generating C introspection for ROS interfaces
No terminal defined for 'u' at line 1 col 1      

uint64 timestamp	# time since system sta
^

Expecting: {'CONST', 'MODULE', 'ENUM', 'STRUCT', 'TYPEDEF', '__ANON_9', 'AT'}
 /home/nuno/PX4/px4_ros_com_ros2/src/px4_msgs/msg/ActuatorArmed.msg
Error processing idl file: /home/nuno/PX4/px4_ros_com_ros2/src/px4_msgs/msg/ActuatorArmed.msg
Traceback (most recent call last):               
  File "/opt/ros/dashing/lib/rosidl_generator_dds_idl/rosidl_generator_dds_idl", line 41, in <module>
    sys.exit(main())
  File "/opt/ros/dashing/lib/rosidl_generator_dds_idl/rosidl_generator_dds_idl", line 36, in main
    args.additional_service_templates,
  File "/opt/ros/dashing/lib/python3.6/site-packages/rosidl_generator_dds_idl/__init__.py", line 51, in generate_dds_idl
    generate_files(generator_arguments_file, mapping, additional_context, keep_case=True)
  File "/home/nuno/PX4/rosidl_ws/install/rosidl_cmake/lib/python3.6/site-packages/rosidl_cmake/__init__.py", line 94, in generate_files
    raise(e)
  File "/home/nuno/PX4/rosidl_ws/install/rosidl_cmake/lib/python3.6/site-packages/rosidl_cmake/__init__.py", line 73, in generate_files
    idl_file = parse_idl_file(locator)
  File "/home/nuno/PX4/rosidl_ws/install/rosidl_parser/lib/python3.6/site-packages/rosidl_parser/parser.py", line 64, in parse_idl_file
    content = parse_idl_string(string, png_file=png_file)
  File "/home/nuno/PX4/rosidl_ws/install/rosidl_parser/lib/python3.6/site-packages/rosidl_parser/parser.py", line 72, in parse_idl_string
    tree = get_ast_from_idl_string(idl_string)
  File "/home/nuno/PX4/rosidl_ws/install/rosidl_parser/lib/python3.6/site-packages/rosidl_parser/parser.py", line 89, in get_ast_from_idl_string
    return _parser.parse(idl_string)
  File "/usr/lib/python3/dist-packages/lark/lark.py", line 306, in parse
    return self.parser.parse(text, start=start)
  File "/usr/lib/python3/dist-packages/lark/parser_frontends.py", line 182, in parse
    return self._parse(text, start)
  File "/usr/lib/python3/dist-packages/lark/parser_frontends.py", line 54, in _parse
    return self.parser.parse(input, start, *args)
  File "/usr/lib/python3/dist-packages/lark/parsers/earley.py", line 293, in parse
    self._parse(stream, columns, to_scan, start_symbol)
  File "/usr/lib/python3/dist-packages/lark/parsers/xearley.py", line 137, in _parse
    to_scan = scan(i, to_scan)
  File "/usr/lib/python3/dist-packages/lark/parsers/xearley.py", line 114, in scan
    raise UnexpectedCharacters(stream, i, text_line, text_column, {item.expect.name for item in to_scan}, set(to_scan))
lark.exceptions.UnexpectedCharacters: No terminal defined for 'u' at line 1 col 1

uint64 timestamp	# time since system sta
^

Expecting: {'CONST', 'MODULE', 'ENUM', 'STRUCT', 'TYPEDEF', '__ANON_9', 'AT'}

Additional information

This follows up the problem reported in ros2/rmw_fastrtps#251 (comment) and in eProsima/Fast-DDS#829. Letting rosidl_generate_interfaces() generate the IDL files and respective typesupport results in having typenames which are not compatible with the ones generated by fastrtpsgen. Previously in Crystal, using rosidl_generate_dds_interfaces(), one was able to generate IDL's and typesupport compatible with the ones from fastrtpsgen (more context in eProsima/Fast-DDS#829).

I am not sure at this point if using rosidl_generate_dds_interfaces() in Dashing is going to result on the same problem or not.

[TSC21]

@dirk-thomas @mjcarroll any inputs to solve this are welcomed. Thank you!

[dirk-thomas]

The error message indicates that you are trying to pass a .msg to a function which expects .idl files.

The function rosidl_generate_dds_interfaces() does not do want you are trying to use it for. It takes already .idl files as input and only transforms them into other .idl files which are compatible with different vendors because some don't support all types defined in IDL 4.2 which are used by ROS.

I am not sure what to are trying to do but you package should probably just invoke the same function as any other message package: rosidl_generate_interfaces().

[dirk-thomas]

That means that the behavior between ROS distros for the same function has changed then. Because I use rosidl_generate_interfaces() in Crystal with msgs as an input and it does generate IDLs from them, so the functionality has definetly changed from one distro to another

I am not sure I understand. Maybe you can describe what you think has changed in more detail.

[TSC21]

The error message indicates that you are trying to pass a .msg to a function which expects .idl files.

The function rosidl_generate_dds_interfaces() does not do want you are trying to use it for. It takes already .idl files as input and only transforms them into other .idl files which are compatible with different vendors because some don't support all types defined in IDL 4.2 which are used by ROS.

I am not sure what to are trying to do but you package should probably just invoke the same function as any other message package: rosidl_generate_interfaces().

That means that the behavior between ROS distros for the same function has changed then. Because I use rosidl_generate_interfaces() in Crystal with msgs as an input and it does generate IDLs from them, so the functionality has definetly changed from one distro to another

[TSC21]

The error message indicates that you are trying to pass a .msg to a function which expects .idl files.
The function rosidl_generate_dds_interfaces() does not do want you are trying to use it for. It takes already .idl files as input and only transforms them into other .idl files which are compatible with different vendors because some don't support all types defined in IDL 4.2 which are used by ROS.
I am not sure what to are trying to do but you package should probably just invoke the same function as any other message package: rosidl_generate_interfaces().

That means that the behavior between ROS distros for the same function has changed then. Because I use rosidl_generate_interfaces() in Crystal with msgs as an input and it does generate IDLs from them, so the functionality has definetly changed from one distro to another

Example: https://github.com/PX4/px4_msgs/blob/master/CMakeLists.txt#L54-L58. This works correctly to all distros before Dashing. As you can see in the Crystal release, https://github.com/ros2/rosidl_dds/blob/crystal/rosidl_generator_dds_idl/cmake/rosidl_generate_dds_interfaces.cmake#L53, so it expected a list of msgs, srvs or actions, not IDL tuples.

[dirk-thomas]

Thanks for the references. I get your point now how the semantic of rosidl_generate_dds_interfaces has changed between Crystal and Dashing. I guess we didn't consider that users would call that API directly and changed the semantic when introducing IDL in Dashing.

I am not sure of it makes sense to attempt to recreate an API which takes .msg files and produces the output of rosidl_generate_dds_interfaces(). I would instead suggest that you try to switch to call rosidl_generate_interfaces() instead and use the .idl files it generates.

[dirk-thomas]

I would instead suggest that you try to switch to call rosidl_generate_interfaces() instead and use the .idl files it generates.

Where you able to use that approach? If not, why didn't it work for you?

[TSC21]

I would instead suggest that you try to switch to call rosidl_generate_interfaces() instead and use the .idl files it generates.

Where you able to use that approach? If not, why didn't it work for you?

Because of what's being reported in eProsima/Fast-DDS#829. The type names are different, so it doesn't work for intra-vendor (Fast-RTPS) communication between two participants in the same domain.

[TSC21]

Although the reported problem is still true (since the API was changed), I am moving away from this since this doesn't seem that is going to be fixed any time soon. Closing.