Calculate the size of the specified directory (include subdirectory)

[Barry-Xu-2018]

rcutils_calculate_directory_size() only support calculating the size of directory without recursive.
Why rcutils_calculate_directory_size() doesn't support recursive ?
Is there any cause ?

[fujitatomoya]

I don't think of anything but i believe that it would be nice if it works recursively.

rcutils_calculate_directory_size is currently used to calculate rosbag2 metadata. since all files in the directory, it does not have to be recursive for right now.

[Barry-Xu-2018]

rcutils_calculate_directory_size is currently used to calculate rosbag2 metadata. since all files in the directory, it does not have to be recursive for right now.

Yes. Now only metadata_io.cpp in rosbag2 call this function.

[fujitatomoya]

it explicitly describes that it does not support recursive search. do we need it to be recursive with actual use cases?

[Barry-Xu-2018]

it explicitly describes that it does not support recursive search. do we need it to be recursive with actual use cases?

Yes. Current implementation in rosbag2 as below

https://github.com/ros2/rosbag2/blob/700817c42f1bbc26ced79aca2186094350c8d1be/rosbag2_storage/src/rosbag2_storage/metadata_io.cpp#L229-L240

It only calculates total of file size in specified path. For sqlite3 database (Now rosbag2 only provide this as storage back-end), it is okay since sqlite3 database is one file. But for others (such as leveldb), it use a directory to save data. Above codes cannot get correct size of database.
I want to update this function to support recursive.
This modification should not introduce the problem for current codes.

[clalancette]

I want to update this function to support recursive.
This modification should not introduce the problem for current codes.

Yeah, I don't see a problem with enhancing it to do that. We may want to introduce a recursive boolean parameter to the function in case the caller really doesn't want that, but otherwise it sounds good to me.

[fujitatomoya]

@clalancette
CC: @Barry-Xu-2018

thanks for the comment!
actually we've been trying to use leveldb as rosbag2 backend database. this enhancement would be necessary to do that.

[Barry-Xu-2018]

@clalancette @fujitatomoya

We may want to introduce a recursive boolean parameter to the function in case the caller really doesn't want that, but otherwise it sounds good to me.

If add a new parameter, after rcutils is updated, we should update rosbag2 at the same time. Otherwise, rosbag2 build failed.

I have tested on Linux platform. But it needs to be tested on window platform.
Now I am preparing window environment.

[fujitatomoya]

If add a new parameter, after rcutils is updated, we should update rosbag2 at the same time. Otherwise, rosbag2 build failed.

i would not like this approach either. we can keep the interface and enhance the functionality to search recursively. if any specific use cases for not recursive search, we could add new function for recursive case.

[Barry-Xu-2018]

i would not like this approach either. we can keep the interface and enhance the functionality to search recursively. if any specific use cases for not recursive search, we could add new function for recursive case.

Yeah. Agree.

@clalancette

How about we enhance rcutils_calculate_directory_size without new parameter ?
Currently, rosbag2 can correctly work with enhanced rcutils_calculate_directory_size().

[clalancette]

How about we enhance rcutils_calculate_directory_size without new parameter ?
Currently, rosbag2 can correctly work with enhanced rcutils_calculate_directory_size().

The reason I'm worried about this is that there is some talk among users of storing "additional" information inside the rosbag2 directory. If they end up storing a lot of additional data in subdirectories, that could throw off the size that rosbag2 is expecting.

If we don't want to break API, we could also add another function that is rcutils_calculate_directory_size_recursive().

pinging @Karsten1987 for insight as to whether always doing a recursive search in rcutils_calculate_directory_size() will cause problems for rosbag2.

[Barry-Xu-2018]

@clalancette

I understand your concerns.
I have checked codes. Only below codes in rosbag2 use this function.

https://github.com/ros2/rosbag2/blob/700817c42f1bbc26ced79aca2186094350c8d1be/rosbag2_storage/src/rosbag2_storage/metadata_io.cpp#L229-L240

After change codes, I also try rosbag2 with default sqlite3 storage plugin and not find the problem.
Of course, we should hear @Karsten1987‘s opinion.

[fujitatomoya]

@Barry-Xu-2018

i guess that you can close this one.

[Barry-Xu-2018]

#306 was merged. So close this issue.