-
Notifications
You must be signed in to change notification settings - Fork 260
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use middleware send and receive timestamps from message_info during recording #1531
Conversation
Note, this was made possible by the merge of ros2/rclcpp#1928 |
CC: @MichaelOrlov |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jmachowinski @fujitatomoya Not this way, and unfortunately not so easy.
We have reserved special fields publishTime
and sequence
for mcap
storage format specifically for this use case. However, currently they are populated with dummy data.
rosbag2/rosbag2_storage_mcap/src/mcap_storage.cpp
Lines 752 to 757 in 195e406
mcap_msg.sequence = 0; | |
if (msg->time_stamp < 0) { | |
RCUTILS_LOG_WARN_NAMED(LOG_NAME, "Invalid message timestamp %ld", msg->time_stamp); | |
} | |
mcap_msg.logTime = mcap::Timestamp(msg->time_stamp); | |
mcap_msg.publishTime = mcap_msg.logTime; |
From mcap vendor Message
struct
/**
* @brief An optional sequence number. If non-zero, sequence numbers should be
* unique per channel and increasing over time.
*/
uint32_t sequence;
/**
* @brief Nanosecond timestamp when this message was recorded or received for
* recording.
*/
Timestamp logTime;
/**
* @brief Nanosecond timestamp when this message was initially published. If
* not available, this should be set to `logTime`.
*/
Timestamp publishTime;
The proper implementation will be to add those two fields publish_time
and sequence_number
to the rosbag2_storage::SerializedBagMessage
struct
rosbag2/rosbag2_storage/include/rosbag2_storage/serialized_bag_message.hpp
Lines 27 to 32 in 195e406
struct SerializedBagMessage | |
{ | |
std::shared_ptr<rcutils_uint8_array_t> serialized_data; | |
rcutils_time_point_value_t time_stamp; | |
std::string topic_name; | |
}; |
and change or add a new rosbag2_cpp::Writer::write(..) API
rosbag2/rosbag2_cpp/include/rosbag2_cpp/writer.hpp
Lines 185 to 200 in 195e406
* Write a serialized message to a bagfile. | |
* The topic will be created if it has not been created already. | |
* | |
* \warning after calling this function, the serialized data will no longer be managed by message. | |
* | |
* \param message rclcpp::SerializedMessage The serialized message to be written to the bagfile | |
* \param topic_name the string of the topic this messages belongs to | |
* \param type_name the string of the type associated with this message | |
* \param time The time stamp of the message | |
* \throws runtime_error if the Writer is not open. | |
*/ | |
void write( | |
std::shared_ptr<const rclcpp::SerializedMessage> message, | |
const std::string & topic_name, | |
const std::string & type_name, | |
const rclcpp::Time & time); |
to be able to pass those two new fields from the rosbag2 transport layer. i.g. recorder.
af4592e
to
a2f06c8
Compare
Do we really need the sequence_number ? I don't see any practical value in it, though I might be missing something... |
Updated the patch. @MichaelOrlov I also started to implement your suggestion, of passing the send timestamp along. The reason for this suggestion would be, that none of the tooling and the API seems to be set up to use the send timestamp, but by switching the timestamps, we could 'hack' our way around this shortcoming. |
@jmachowinski As reagards:
The sequence number needed to be able to detect on what step messages were lost and how many of them were lost. Or the other way around to make sure that it wasn't data losses during recording. i.e. check for data consistency. As regards:
No hacking solutions will be accepted for the rosbag2. Sorry, I have a strong opinion on that. |
i think use case of i guess, we can generate incremental number for |
I also got a strong opinion against workarounds. But in this case I don't see a good way around it. The problem is, that its not only this library. We got rqt_bag and foxglove studio down the line, that would need patches as well. For our use case the receive timestamp is mostly useless and we are way more interested in analyzing the data in send order. And I guess we are not the only one... This leads to a more general question, is the receive timestamp actually relevant to anyone ?
Okay, that is a valid application.
@fujitatomoya The technical part on how to fill in the sequence number is clear to me, its more about is it worth the memory overhead. Adding a uint32_t might not sound like much, but for comparison, we have a own implementation, of the snapshot feature, which has a memory food print that is about ~1/3 smaller compared to rosbag2, as we don't save the topic name but a 16 bit topic id in every message. |
@jmachowinski As regards to the:
It only means that if other tools like As regards
Well, at least currently everybody else using the receive timestamp and changing it by default to the published timestamp may cause breakage in workflow and other algorithms. The safest way will be to add an option with the new fields and new functionality. As regards to the:
This is not true with MCAP file format anymore because:
|
I am talking about RAM usage, not SSD/HDD usage. The problem is within SerializedBagMessage that is stored in CircularMessageCache while using the snapshot feature.
I agree that it is the safest route, that is out of the question. I am more wondering, if it was a deliberate decision to use the receive timestamp in the first place, or was it just used because it was available. |
good question. |
@jmachowinski @fujitatomoya
I also don't know the answer to this question. It was before me. Another valuable point that was likely driving the decision is clock synchronization on different ECUs and nodes. @jmachowinski As regards:
Noted. As mitigation for such a problem, I only can suggest to start using |
This pull request has been mentioned on ROS Discourse. There might be relevant details there: https://discourse.ros.org/t/ros-2-tsc-meeting-minutes-for-2024-01-18/35779/1 |
aa55430
to
74341d2
Compare
I added the publisher timestamp. And renamed the existing one to receive timestamp. The sequence number is a different beast, and I would put it into a separate pull request. The second available sequence number is the one of the sender, which would tell you that you lost messages in transmission, but it will jump all over the place if you got multiple senders. Don't have an idea how to solve this... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jmachowinski Thanks for your contribution.
I think it does make sense to use rcutils_timestamp_value_t
instead of the rclcpp::Time
in the newly added write(..)
API for efficiency. See explanations and details in the dedicated comment.
Also, I would like to ask a renames for the time stamp variables. See motivation and explanation in the dedicated comment.
rosbag2_compression/src/rosbag2_compression/sequential_compression_writer.cpp
Outdated
Show resolved
Hide resolved
rosbag2_storage/include/rosbag2_storage/serialized_bag_message.hpp
Outdated
Show resolved
Hide resolved
rosbag2_transport/test/rosbag2_transport/test_record_all_use_sim_time.cpp
Outdated
Show resolved
Hide resolved
@jmachowinski As regards
Yeah, that is not easy. We tried to brainstorm a situation around the sequence number during the last typedef struct RMW_PUBLIC_TYPE rmw_gid_s
{
/// Name of the rmw implementation
const char * implementation_identifier;
/// Byte data GID value
uint8_t data[RMW_GID_STORAGE_SIZE];
} rmw_gid_t; where After some more consideration, it seems I found an elegant and I think proper solution. As regards the current implementation. |
6fe4592
to
f7f7a51
Compare
@MichaelOrlov adopted code to the request changes |
@jmachowinski @clalancette I've found the place where it is settled up to zero. rmw_connextdds_message_info_from_dds(
rmw_message_info_t * const to,
const DDS_SampleInfo * const from)
{
rmw_connextdds_ih_to_gid(from->publication_handle, to->publisher_gid);
// Message timestamps are disabled on Windows because RTI Connext DDS
// does not support a high enough clock resolution by default (see: _ftime()).
#if !RTI_WIN32
to->source_timestamp = dds_time_to_u64(&from->source_timestamp);
to->received_timestamp = dds_time_to_u64(&from->reception_timestamp);
#else
to->source_timestamp = 0;
to->received_timestamp = 0;
#endif /* !RTI_WIN32 */ We need either a workaround for it or a fix in the |
@MichaelOrlov Should I add a workaround ? I would go with an ifdef in the recorder.cpp that would disable this feature if RTI_WIN32 is set. |
@jmachowinski Yes, please. However, I am not sure that Please note that we need to fallback to the old behavior in recorded if |
@MichaelOrlov added a workaround, that disables this feature on windows. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jmachowinski It turns out that in this workaround we are falling back to the old behavior for all RMWs under the Windows.
However, we need to make an exclusion only for the RTI Connext DDS.
Perhaps we can use following if construct to detect it
if (std::string(rmw_get_implementation_identifier()).find("rmw_connextdds") != 0) {
Also, will need to adjust a newly added test for this workaround as well.
c954ea7
to
76c351e
Compare
I was under the impression, that we don't have any rwm timestamp support on windows, regardless of the DDS implementation. I updated the patch, to only blacklist connextdds. I'll have a look at the test later. |
This pull request has been mentioned on ROS Discourse. There might be relevant details there: https://discourse.ros.org/t/ros-2-tsc-meeting-minutes-for-2024-03-21/36814/1 |
… case This should create better timestamp in cases of high load. Signed-off-by: Janosch Machowinski <[email protected]>
…ime_stamp to receive_time_stamp Signed-off-by: Janosch Machowinski <[email protected]>
…amp to send_timestamp Co-authored-by: Michael Orlov <[email protected]> Signed-off-by: Janosch Machowinski <[email protected]>
Signed-off-by: Janosch Machowinski <[email protected]>
Signed-off-by: Janosch Machowinski <[email protected]>
…extdds. The windows rmw connextdds does not provide a value in source timestamp nor in receive timestamp, therefore we fall back to current node time. Signed-off-by: Janosch Machowinski <[email protected]>
Signed-off-by: Janosch Machowinski <[email protected]>
76c351e
to
1f5f938
Compare
Signed-off-by: Janosch Machowinski <[email protected]>
1f5f938
to
7154b5a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jmachowinski Thanks! LGTM.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jmachowinski Sorry. Completely forgot.
Can you please regenerate Python stub files (.pyi)?
The instruction is in the https://github.com/ros2/rosbag2/blob/rolling/rosbag2_py/README.md
Gist: https://gist.githubusercontent.com/MichaelOrlov/7fafb68b299a4679142305c2a3cf6c99/raw/02674aa34517082806b39b18fdf48b8cbb37791d/ros2.repos |
Signed-off-by: Janosch Machowinski <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM with green CI
Signed-off-by: Mikael Arguedas <[email protected]>
This should create a better timestamp in cases of high load.
Note, I am unsure, if we should use the received_timestamp or if the source_timestamp would be the better option.
Disclaimer: The
send_timestamp
will be available only for theMCAP
storage format for a while, since we already have a special field for it inMCAP
format specification.