Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go: fix channel auto coalescing in merge #1322

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

bradsquicciarini-coco
Copy link
Contributor

Changelog

Ensure channel auto coalescing is deterministic in mcap merge

Docs

Description

In the mcap merge command, the channel auto coalescing is currently non-deterministic due to the random ordering of iterating over a map in go. This change just sorts the metadata keys before we hash it to ensure for the same channel metadata we always get the same hash.

For example, if trying to merge these to mcaps

library:   mcap go v1.7.1; mcap go v1.7.0
profile:
messages:  800
duration:  19.979441843s
start:     2024-07-01T10:23:52.441156385-07:00 (1719854632.441156385)
end:       2024-07-01T10:24:12.420598228-07:00 (1719854652.420598228)
compression:
        zstd: [1/1 chunks] [273.00 KiB/36.78 KiB (86.53%)] [1.84 KiB/sec]
channels:
        (4) /imu  800 msgs (40.04 Hz)   : sensor_msgs/Imu [ros1msg]
channels: 1
attachments: 0
metadata: 0
library:   mcap go v1.7.1; mcap go v1.7.0
profile:
messages:  800
duration:  19.979921296s
start:     2024-07-01T15:05:18.571434739-07:00 (1719871518.571434739)
end:       2024-07-01T15:05:38.551356035-07:00 (1719871538.551356035)
compression:
        zstd: [1/1 chunks] [273.00 KiB/35.08 KiB (87.15%)] [1.75 KiB/sec]
channels:
        (4) /imu  800 msgs (40.04 Hz)   : sensor_msgs/Imu [ros1msg]
channels: 1
attachments: 0
metadata: 0

will sometimes give this

$ mcap merge imu0.mcap imu1.mcap -o test.mcap && mcap info test.mcap
library:   mcap go v1.7.1
profile:
messages:  1600
duration:  4h41m46.11019965s
start:     2024-07-01T10:23:52.441156385-07:00 (1719854632.441156385)
end:       2024-07-01T15:05:38.551356035-07:00 (1719871538.551356035)
compression:
        zstd: [1/1 chunks] [543.32 KiB/67.37 KiB (87.60%)] [4.00 B/sec]
channels:
        (1) /imu  1600 msgs (0.09 Hz)   : sensor_msgs/Imu [ros1msg]
channels: 1
attachments: 0
metadata: 0

and sometimes give this

mcap merge imu0.mcap imu1.mcap -o test.mcap && mcap info test.mcap
library:   mcap go v1.7.1
profile:
messages:  1600
duration:  4h41m46.11019965s
start:     2024-07-01T10:23:52.441156385-07:00 (1719854632.441156385)
end:       2024-07-01T15:05:38.551356035-07:00 (1719871538.551356035)
compression:
        zstd: [1/1 chunks] [543.45 KiB/67.85 KiB (87.51%)] [4.00 B/sec]
channels:
        (1) /imu  800 msgs (0.05 Hz)   : sensor_msgs/Imu [ros1msg]
        (2) /imu  800 msgs (0.05 Hz)   : sensor_msgs/Imu [ros1msg]
channels: 2
attachments: 0
metadata: 0

I've updated the unit tests to test for this.

Fixes: foxglove/repo#772

@bradsquicciarini-coco bradsquicciarini-coco force-pushed the bugfix/cli-mcap-merge-metadata-hash-order branch from 68ae983 to 195946b Compare January 26, 2025 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

1 participant