Error: 'charmap' codec can't decode byte 0x8f in position 156

[minggangw]

Bug report

Required Info:

• Operating System:
  • Windows 10 ONLY
• Installation type:
  • Binary of nightly build (http://ci.ros2.org/view/packaging/job/packaging_windows/lastSuccessfulBuild/artifact/ws/ros2-package-windows-AMD64.zip)
• Version or commit hash:
• DDS implementation:
• Client library (if applicable):

Steps to reproduce issue

Pass the path of WStrings.msg as the parameter of Python method parse_message_file

Expected behavior

Execute successfully

Actual behavior

Error: 'charmap' codec can't decode byte 0x8f in position 156: character maps to

Additional information

• Python version: Python3.7 x64
• Suspection: there are some special characters though commented out, and I found possible some potential reason, see https://stackoverflow.com/questions/9233027/unicodedecodeerror-charmap-codec-cant-decode-byte-x-in-position-y-character

Thanks!

[sloretz]

This sounds little bit like ros2/system_tests#362 (comment) . Maybe WStrings.msg needs a byte order mark? Alternatively, maybe parse_message_file() needs to expose an encoding keyword arg for the call to open()?

[minggangw]

Maybe it's kind of Python version of it.

[dirk-thomas]

Any generated source file which contains non-ASCII characters (coming from e.g. default values in the interface file) needs to include a BOM.

[dirk-thomas]

I misread the description and comments and assumed the problem was in generated C / C++ code.

Since this actually happens when reading the interface file I think #390 should be sufficient to resolve your problem. @minggangw please comment here if it addressed your problem.

[dirk-thomas]

Reopening since there are follow up issues when using non-ASCII character in e.g. default values:

• add wstring default values with non-ASCII characters test_interface_files#8 adds a message with non-ASCII characters in the default values
• use latin-1 encoding when reading/writing .idl files, prepend BOM to generated C/C++ files when necessary #391 (first commit) fixes the encoding when writing the expanded templates
• fix condition to not skip FastRTPS to FastRTPS pub/sub tests system_tests#377 fixes a regression reenabling the pub/sub FastRTPS tests
• fix encoding handling when writing to stdout and log files launch#280 fixes launch to properly output non-ASCII characters to stdout as well as log files
• improve handling of encoding ament/ament_cmake#181 fixes encoding handling in the test runner / result generation logic
• use latin-1 encoding when reading/writing .idl files, prepend BOM to generated C/C++ files when necessary #391 (second commit) prepends a BOM to generated C/C++ file when necessary

[dirk-thomas]

With all PRs rebased to address other unrelated test failures I triggered an full round of CI builds to double check for regressions:

• Linux (unrelated test failure, also happening in nightly)
• Linux-aarch64
• macOS (FastRTPS pub WStrings to Connext sub fails, skipped by ros2/system_tests@49cdb2f)
• Windows (unrelated test failures)

[minggangw]

This problem is fixed, I verified it on https://ci.appveyor.com/project/minggangw/rclnodejs/builds/26335124, thanks!

[dirk-thomas]

Until the list of pending PRs is merged we will keep this ticket open.

[dirk-thomas]

I disabled one case which doesn't seem to work cross vendor: ros2/system_tests@5d100b6

Hopefully the final round of CI:

• Linux
• Linux-aarch64
• macOS
• Windows (unrelated failures)