Port Multi-weight support from prototype to main TorchVision

[datumbox]

Fixes #4679

How to use

The Multi-weight support API deprecates the old pretrained=True parameter from TorchVision's model builders in favour of a new weights parameter. The implementation is fully BC and allows us to offer multiple pre-trained weights for the same model variant. Moreover it allows us to bundle the inference/preprocessing transforms of the model and its crucial meta-data along with the weights. For more information, check out our original article on PyTorch blog (just make sure to replace torchvision.prototype.models with torchvision.models).

Example:

from PIL import Image
from torchvision.models import ResNet50_Weights, resnet50

img = Image.open("test/assets/encode_jpeg/grace_hopper_517x606.jpg")

weights = ResNet50_Weights.IMAGENET1K_V2
model = resnet50(weights=weights).eval()
preprocess = weights.transforms()

batch = preprocess(img).unsqueeze(0)
prediction = model(batch)

---

This PR includes the following work

• Ports all models from prototype to main area and adds documentation to the builders
• Updates the docs, examples, tutorials and galleries
• Updates the test_models.py to use weights and move the test_prototype_models.py tests
• Updates the reference scripts and me _presets.* private
• Fix BC breakings on weights_backbone
• Test that all pretrained models are not affected and their builders work well

Checks using existing tests

To confirm that all model builders work as expected:

• We run the tests of all models on the same expected files as on master. All tests pass for untrained models.
• We modify the test_models.py on the main branch to estimate new expected files with pretrained=True (see [NOMERGE] Expected values on pre-trained models (main branch) #5655). Then we run the tests on this branch against the new expected files. All tests pass for trained models.

Checks using reference scripts

To confirm that all the reference scripts still work as expected we ran them all and compared their results with existing model numbers:

torchrun --nproc_per_node=2 train.py --model efficientnet_b0 --test-only --weights EfficientNet_B0_Weights.IMAGENET1K_V1
Test:  Acc@1 77.684 Acc@5 93.536

torchrun --nproc_per_node=2 train.py --model retinanet_resnet50_fpn --test-only --weights RetinaNet_ResNet50_FPN_Weights.COCO_V1
IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.363
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.557
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.382
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.193
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.400
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.490
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.314
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.500
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.540
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.340
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.581
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.696

torchrun --nproc_per_node=2 train.py --model lraspp_mobilenet_v3_large --test-only --weights LRASPP_MobileNet_V3_Large_Weights.COCO_WITH_VOC_LABELS_V1
global correct: 91.2
average row correct: ['94.5', '84.3', '69.5', '72.8', '57.7', '42.0', '77.0', '56.9', '90.4', '36.1', '76.0', '60.8', '81.4', '78.9', '81.0', '87.6', '51.3', '83.9', '62.2', '84.2', '56.1']
IoU: ['90.2', '69.2', '57.7', '58.5', '47.8', '35.7', '69.5', '47.1', '79.1', '29.6', '62.6', '34.2', '65.5', '63.4', '70.0', '76.8', '30.1', '61.9', '46.8', '70.6', '49.1']
mean IoU: 57.9

torchrun --nproc_per_node 1 --nnodes 1 train.py --val-dataset sintel --batch-size 1 --dataset-root $dataset_root --model raft_large --weights Raft_Large_Weights.C_T_SKHT_V2
Batch-processed 1041 / 1041 samples. Going to process the remaining samples individually, if any.
Sintel val clean epe: 0.6499	1px: 0.9255	3px: 0.9701	5px: 0.9805	per_image_epe: 0.6499	f1: 2.6805
Batch-processed 1041 / 1041 samples. Going to process the remaining samples individually, if any.
Sintel val final epe: 1.0197	1px: 0.8898	3px: 0.9493	5px: 0.9660	per_image_epe: 1.0197	f1: 4.5752

torchrun --nproc_per_node=8 train.py --data-path /datasets01/kinetics/070618/400/ --train-dir=val --val-dir=val --batch-size=16 --sync-bn --test-only --weights R2Plus1D_18_Weights.KINETICS400_V1 --cache-dataset
 * Clip Acc@1 56.488 Clip Acc@5 77.745

The difference on the video accuracy comparing to the docs is not related to this PR. It pre-existed on main and was first reported at #4839

[NicolasHug]

Thanks a lot @datumbox . I made some minor comments but LGTM anyway.

[NicolasHug]

(Replying to https://github.com/pytorch/vision/pull/5618/files#r829105451 here because GitHub hides the .yml file)

Are these tests really that long to run? If they are, could we simply just skip them for all non-linux CI jobs (we could even pin a Python version if we want to make sure they only run once)? No strong opinion but perhaps there are simpler solutions than adding a new job

[datumbox]

They don't take that long for now and yes we could skip them on all but 1 configuration. One potential benefit from doing it this way is we don't have to run the tests on FBcode where we would have to download the weights serially from manifold.

Let's review the model testing strategy as a whole later this half and redesign our testing strategy. We can catch work together on this if you have the bandwidth.

[jdsgomes]

Just left a clarification let me know your thoughts.

[jdsgomes]

LGTM! thank you

[datumbox]

@NicolasHug @jdsgomes Thank you both for your heroic reviews and support on this work. I'm running few last checks on the references and if all pass, I'll merge the PR soon.

I really hope that going forwards we will find better ways to land prototype APIs into main without such massive PRs.

@zhiqwang @oke-aditya I'm looking forward to your input on how to improve this work prior the release. If you spot anything weird while testing main please tag me to fix. PRs are very welcome. :)

[datumbox]

All tests pass. The failing jobs for binary_linux_conda_py*_cu115 are unrelated and they are being tackled at #5644. Merging...

[oke-aditya]

Vamos!! Multi Weight support.

( I really wish if GitHub has GIF search like Giphy lol)

I will give this a try and see if I could make any improvements