[RFC] Moving MXNet-AMP to core

[mk-61]

MXNet already has experimental AMP (Automatic Mixed Precision) support, exposed in mxnet.contrib package. It is used for automatic casting models to both float16 and bfloat16. This RFC covers moving it into core / making a first-class feature, as well as further development.

Here's a rough task break down for the initial move:

• Need to ensure AMP works with numpy ops - i.e., all ops are in either of the lists - done in AMP support for Numpy ops #19036
• API change: make loss scale public (Make loss scale public in AMP #17507) - done in AMP support for Numpy ops #19036
• Transparent / lazy AMP initialization? (Got "kFlag == type_flag_: TBlob.get_with_shape: data type do not match specified type.Expected: 0 v.s. given 2" when training with amp. #18902 (comment)) - a warning added, when amp.init() is called and a model already exists in AMP support for Numpy ops #19036
• A number of issues has to be resolved to improve user experience:
  1. Cannot load trainer with AMP (Cannot load trainer with AMP #16858) - fixed in Get rid of monkey patching in LossScaler overflow handling #18959
  2. There's a CUDA crash (IMA) in amp_multicast, happens on some models (Yolo3) - fixed in Fix possible IMA in amp_multicast fusion #19318
  3. AMP not reusing weights on recursive networks (AMP not reusing weights on recursive networks #19019)
• The actual shuffling code around and updating import paths

Post move:

1. Layout optimization - upstreaming feature already existing in NVIDIA NGC container. This helps convolutions' performance by automatically casting between NCHW and NHWC layouts.
2. Explore alternatives to front end ops monkey-patching (AMP for mx2 #18697)
3. Add a way for the user to turn AMP off, and to control AMP setting via a context manager.

[github-actions]

Welcome to Apache MXNet (incubating)! We are on a mission to democratize AI, and we are glad that you are contributing to it by opening this issue.
Please make sure to include all the relevant context, and one of the @apache/mxnet-committers will be here shortly.
If you are interested in contributing to our project, let us know! Also, be sure to check out our guide on contributing to MXNet and our development guides wiki.

[szha]

cc @sxjscience @leezu

[sxjscience]

@mk-61 If you'd like to see some test cases of the new numpy API, you can also try the numpy version of GluonNLP: https://github.com/dmlc/gluon-nlp/tree/numpy . Would we connect via Slack?