[Feature] Loading objects from different backends and dumping objects to different backends

[zhouzaida]

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

The motivation of the PR is to load objects from different backends and dump objects to different backends but do not need to modify the existing interface.

It includes three parts:

• Refactor FileClient to support instantiating it with the prefix of filepath.
• Refactor the interface of StorageBackend
• Refactor IO helper functions to support loading objects from different backends and dumping objects to different backends.

Modification

1. FileClient

   • Add a new parameter prefixes to register_backend, but the parameter is optional
   • Add parse_uri_prefix static method to parse the prefix of a filepath
   • Add get_local_path to download a file from a remote server
   • Add put, put_text, remove, exists, isdir, isfile, concat_paths
   • Add __new__ and use the singleton pattern to avoid creating duplicate objects

2. StorageBackend

   • Implement eigth methods put , put_text, remove, exists, isdir, isfile, concat_paths, get_local_path and list_dir_or_file for PetrelBackend
   • Implement nine methods put , put_text, remove, exists, isdir, isfile, concat_paths, get_local_path and list_dir_or_file for HardDiskBackend

3. IO helper functions
   file_client_args is a parameter to instantiate a FileClient. If file_client_args is None, the prefix of the filepath will be used to instantiate a FileClient.

   • Add a new parameter file_client_args to load which makes it support loading object from different backends
   • Add a new parameter file_client_args to dump which makes it support dumping object to different backends
   • Add a new parameter file_client_args to list_from_file which make it support parsing file from different backends
   • Add a new parameter file_client_args to dict_from_file which make it support parsing file from different backends

BC-breaking (Optional)

No

Use cases (Optional)

1. Instantiating FileClient with the prefix of filepath

   http_path = 'http://path/of/your/file'
   prefix = FileClient.parse_uri_prefix(http_path)
   client = FileClient(prefix=prefix)
   client = FileClient.infer_client(uri=http_path)
   s3_path = 's3://your_bucket/img.png'
   prefix = FileClient.parse_uri_prefix(http_path)
   client = FileCilent(prefix=prefix)  
   client = FileClient.infer_client(uri=s3_path)

2. Loading objects from different backends and dumping objects to different backends

   http_path = 'http://path/of/your/file'
   mmcv.load(http_path)
   s3_path = 's3://your_bucket/img.png'
   mmcv.load(s3_path)

Test

• list_dir_or_file of FileClient works as expected
• get_local_path of FileClient works as expected
• Other interfaces of FileClient can be tested at [Feature] Upload checkpoints and logs to ceph #1375

Checklist

Before PR:

• I have read and followed the workflow indicated in the CONTRIBUTING.md to create this PR.
• Pre-commit or linting tools indicated in CONTRIBUTING.md are used to fix the potential lint issues.
• Bug fixes are covered by unit tests, the case that causes the bug should be added in the unit tests.
• New functionalities are covered by complete unit tests. If not, please add more unit test to ensure the correctness.
• The documentation has been modified accordingly, including docstring or example tutorials.

After PR:

• CLA has been signed and all committers have signed the CLA in this PR.
• If the modification has potential influence on downstream or other related projects, this PR should be tested with some of those projects, like MMDet or MMCls.

[codecov]

Codecov Report

Merging #1330 (d4b6d96) into master (8cac7c2) will increase coverage by 0.08%.
The diff coverage is 86.76%.

❗ Current head d4b6d96 differs from pull request most recent head 6987038. Consider uploading reports for the commit 6987038 to get more accurate results

@@            Coverage Diff             @@
##           master    #1330      +/-   ##
==========================================
+ Coverage   68.59%   68.67%   +0.08%     
==========================================
  Files         164      164              
  Lines       10891    10996     +105     
  Branches     1991     2008      +17     
==========================================
+ Hits         7471     7552      +81     
- Misses       3030     3049      +19     
- Partials      390      395       +5     

Flag	Coverage Δ
unittests	68.67% <86.76%> (+0.08%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmcv/fileio/file_client.py	85.64% <83.33%> (-4.43%)	⬇️
mmcv/fileio/handlers/base.py	53.33% <100.00%> (-25.24%)	⬇️
mmcv/fileio/handlers/pickle_handler.py	87.50% <100.00%> (-12.50%)	⬇️
mmcv/fileio/io.py	76.56% <100.00%> (+6.56%)	⬆️
mmcv/fileio/parse.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2d73eaf...6987038. Read the comment docs.

[ZwwWayne]

Can be merged after resolving final comments.