Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add LR Scheduler #4694

Merged
merged 7 commits into from
Aug 4, 2020
Merged

Add LR Scheduler #4694

merged 7 commits into from
Aug 4, 2020

Conversation

rayankrish
Copy link
Contributor

  • Note: should be merged after train step (Add train_step #4677)
  • Implements LR Scheduler
  • Created a separate pull request with isolated commits for this task

@rayankrish rayankrish requested a review from a team as a code owner August 3, 2020 18:59
@thiagocrepaldi thiagocrepaldi linked an issue Aug 3, 2020 that may be closed by this pull request
[WIP] New PyTorch frontend API #4176
Closed
@thiagocrepaldi thiagocrepaldi changed the title Frontend/feature/lr scheduler Add LR Scheduler Aug 3, 2020
rayankrish
rayankrish commented Aug 3, 2020
View reviewed changes
orttraining/orttraining/test/python/orttraining_test_orttrainer_frontend.py Outdated
max_train_step = 1
warmup = 0.5
initial_lr = 1
optim_config = optim.SGDConfig() if not lr_scheduler else optim.SGDConfig(lr=initial_lr)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thiagocrepaldi when lr=1 is set but no lr_scheduler is given, the train step crashes by an input error. This is a temporary fix, but it's unclear what the problem is

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you repro this in the current API/example?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I reproduced the error in two ways.

  • Pass a learning rate of 1 at each train step
  • Pass a learning rate scheduler to ORTTrainer that always returns 1

Error:
RuntimeError: Error in execution: Unexpected input data type. Actual: (N11onnxruntime17PrimitiveDataTypeIlEE) , expected: (N11onnxruntime17PrimitiveDataTypeIfEE)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At orttraining/orttraining/python/training/optim/config.py, around line 47, there is this:

assert (isinstance(defaults['lr'], float) or isinstance(defaults['lr'], int)) and defaults['lr'] >= 0, "lr must be a positive number"

Now that we create an internal IODescription with torch.float32, it is clear that we should change it to

assert isinstance(defaults['lr'], float) and defaults['lr'] >= 0, "lr must be a positive number" to prevent this issue

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rayankrish

thiagocrepaldi
thiagocrepaldi approved these changes Aug 4, 2020
View reviewed changes
Copy link
Contributor

@thiagocrepaldi thiagocrepaldi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, Maybe @liqunfu has some feedback

@thiagocrepaldi thiagocrepaldi merged commit d91ec63 into microsoft:feature/new_pytorch_frontend Aug 4, 2020
@thiagocrepaldi thiagocrepaldi mentioned this pull request Aug 10, 2020
Closed
@thiagocrepaldi thiagocrepaldi removed a link to an issue Aug 10, 2020
[WIP] New PyTorch frontend API #4176
Closed
thiagocrepaldi pushed a commit that referenced this pull request Aug 12, 2020
@rayankrish
Add LR Scheduler (#4694)
bd7a422 
Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
Co-authored-by: Thiago Crepaldi <[email protected]>
thiagocrepaldi pushed a commit that referenced this pull request Aug 14, 2020
@rayankrish
Add LR Scheduler (#4694)
92c5ba4 
Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
Co-authored-by: Thiago Crepaldi <[email protected]>
thiagocrepaldi pushed a commit that referenced this pull request Aug 15, 2020
@rayankrish
Add LR Scheduler (#4694)
1e1653d 
Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
Co-authored-by: Thiago Crepaldi <[email protected]>
thiagocrepaldi pushed a commit that referenced this pull request Aug 15, 2020
@rayankrish
Add LR Scheduler (#4694)
e8269a4 
Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
Co-authored-by: Thiago Crepaldi <[email protected]>
thiagocrepaldi pushed a commit that referenced this pull request Aug 17, 2020
@rayankrish
Add new PytTrch front-end (#4815)
42408aa 
* Add ORTTrainerOptions class for the new pytorch frontend (#4382)

Add ORTTrainerOptions class and some placeholders

* Add _ORTTrainerModelDesc to perform validation for model description (#4416)

* Add Loss Scaler classes to the new frontend (#4306)

* Add TrainStepInfo used on the new frontend API (#4256)

* Add Optimizer classes to the new frontend (#4280)

* Add LRScheduler implementation (#4357)

* Add basic ORTTrainer API (#4435)

This PR presents the public API for ORTTrainer for the short term
development.

It also validates and saves input parameters, which will be used in the
next stages, such as building ONNX model, post processing the model and
configuring the training session

* Add opset_version into ORTTrainerOptions and change type of ORTTrainer.loss_fn (#4592)

* Update ModelDescription and minor fix on ORTTrainer ctor (#4605)

* Update ModelDescription and minor fix on ORTTrainer/ORTTrainerOptions

This PR keeps the public API intact, but changes how model description is stored on the backend

Currently, users creates a dict with two lists of tuples.
One list called 'inputs' and each tuple has the following format tuple(name, shape).
The second list is called 'outputs' and each tuple can be either tuple(name, shape) or tuple(name, shape, is_loss).

With this PR, when this dict is passed in to ORTTrainer, it is fully validated as usual.
However, tuples are internally replaced by namedtuples and all output tuples will have
tuple(name, shape, is_loss) format instead of is_loss being optionally present.

Additionally to that normalization in the internal representation (which eases coding),
two internal methods were created to replace a namedtuple(name, shape) to namedtuple(name, shape, dtype)
or namedtuple(name, shape, is_loss, dtype) dependeing whether the tuple is an input or output.

This is necessary as ORTTRainer finds out data types of each input/output during model export to onnx.

Finally, a minor fix was done on ORTTrainer. It could initialize ORTTrainerOptions incorrectly when options=None

* Rename input name for test

* Add ONNX Model Export to New Frontend (#4612)

Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
Co-authored-by: Thiago Crepaldi <[email protected]>

* Create training session + minor improvements (#4668)

Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>

* Save ONNX model in file (#4671)

Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>

* Add eval step (#4674)

Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>

* Add train_step (#4677)

Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>

* Add LR Scheduler (#4694)

Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
Co-authored-by: Thiago Crepaldi <[email protected]>

* Add deterministic compute tests (#4716)


Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
Co-authored-by: Thiago Crepaldi <[email protected]>

* Add legacy vs experimental ORTTrainer accuracy comparison (#4727)

Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
Co-authored-by: Thiago Crepaldi <[email protected]>

* Add Mixed precision/LossScaler + several fixes (#4739)

Additionally to the mixed precision/loss scaler code, this PR includes:

* Fix CUDA training
* Add optimization_step into TrainStepInfo class
* Refactor LRSCheduler to use optimization_step instead of step
* Updated several default values at ORTTrainerOptions
* Add initial Gradient Accumulation supported. Untested
* Fix ONNX model post processing
* Refactor unit tests

* Add ONNX BERT example + minor fixes (#4757)

* Fix training issue when passing ONNX file into ORTTrainer

Co-authored-by: Thiago Crepaldi <[email protected]>
Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>

* Add Dynamic Shape support (#4758)

* Update DeepSpeed Zero Stage option to a separate option group (#4772)

* Add support to fetches (#4777)

* Add Gradient Accumulation Steps support (#4793)

* Fix Dynamic Axes feature and add unit test (#4795)

* Add frozen weights test (#4807)

* Move new pytorch front-end to 'experimental' namespace (#4814)

* Fix build

Co-authored-by: Rayan-Krishnan <[email protected]>
Co-authored-by: Rayan Krishnan <t-rakr@OrtDevTest2v100.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liqunfu liqunfu liqunfu approved these changes

@thiagocrepaldi thiagocrepaldi thiagocrepaldi approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@rayankrish @liqunfu @thiagocrepaldi