HDDS-12075. Update read replica command to handle slash character in key names

[ptlrs]

What changes were proposed in this pull request?

Key names may include arbitrary number of / characters, such as /volumename/bucketname/test1/test2.

The read-replicas command assumes key names will be single level or not include any / characters. This results in failures when the command is run on multilevel keys.

This PR:

• Updates the ReadReplicas class to create parent directories of key names
• Update robot tests to handle multilevel key names
• Update robot tests to search in directories recursively as the concerned files may be nested and not at the top level of the directory

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-12075

How was this patch tested?

Tested locally by running the robot tests
CI:
https://github.com/ptlrs/ozone/actions/runs/12763363095

[adoroszlai]

Thanks @ptlrs for the patch.

read-replicas produces output into local dir, whose name is calculated like volume_bucket_key_date. It does not create subdirectories for volume and bucket, so I don't think it should create subdirectories if the key name happens to include /. The tool should simply replace any / in the key name with _. I think this would be more consistent with the existing dir naming scheme, and also simplify user experience.

[ptlrs]

Thank you for the reivew @adoroszlai. I had the same thought of replacing / with _ in key names while I was implementing this change. However, I didn't do it because of potential key name conflicts.

For example,
Key rmp/1 will be converted to rmp_1
Key rmp_1 will remain as rmp_1.

While we do use a timestamp suffix to allow multiple runs of read-replicas with the key same name, there is an ambiguity introduced here. We can't readily grep for or reconstruct the original key name from the generated name. We may need to check the contents of the generated manifest file.

Is this a significant inconvenience? If not, then I can make this change.

[adoroszlai]

I didn't do it because of potential key name conflicts.

Thanks @ptlrs for the background on this.

For example, Key rmp/1 will be converted to rmp_1 Key rmp_1 will remain as rmp_1.

This is a debug command, so I think it's OK. Let's wait for others' opinion before making changes to the patch.