OptimWrapper sets same param groups as Optimizer

[warner-benjamin]

This PR harmonizes the default parameter group setting between OptimWrapper and Optimizer by modifying OptimWrapper to match Optimizer's logic.

Currently, when passed the default Learner.splitter of trainable_params, Optimizer creates one parameter group for the entire model but OptimWrapper creates a parameter group for every trainable parameter. For example, ResNet50 results with ~160 parameter groups instead of the expected one parameter group.

Users can provide their own parameter groups by setting convert_groups=False and passing in a dictionary to params as before.

This PR also modifies OptimWrapper.hypers to be compatible with a larger number of PyTorch optimizers by allowing new param group dictionary keys to be added during training. Nvidia's Apex optimizers are one example with this behavior with FusedAdam adding 'step' to each parameter group during training but not on optimizer creation.

This also makes OptimWrapper forward compatible with new param group additions not currently in pytorch_hp_map, such as 'fused' coming in a future PyTorch release.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[warner-benjamin]

I'm reasonably certain the CI error is not related to my PR and it needs to be reran later when the networking issues are resolved.

ERROR: Could not install packages due to an OSError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url:

[jph00]

Many thanks @warner-benjamin !