[MXNET-1255] update hybridize documentation #13597

roywei · 2018-12-10T04:57:05Z

Description

This is the first step to address issues in:
#10875
#12100
#12109

First provide documentation and current work around.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Add documentations for operators does not work with hybridization

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

docs/tutorials/gluon/hybrid.md

roywei · 2018-12-12T00:32:58Z

@mxnet-label-bot add[Doc, pr-awaiting-review]

ChaiBapchya

Pretty handy doc! Thanks for the summarization.

docs/tutorials/gluon/hybrid.md

ChaiBapchya · 2018-12-16T22:11:06Z

docs/tutorials/gluon/hybrid.md

+    return x[0]
+```
+
+The current workaround is explicitly call [`slice`](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.slice) operators in `hybrid_forwar`d.


nit : hybrid_forward

ChaiBapchya · 2018-12-16T22:12:40Z

docs/tutorials/gluon/hybrid.md

+```
+
+### Slice
+[] in NDArray is to get a slice from the array. [] in Symbol is to get an output from a grouped symbol.


nit : how about
[] in NDArray is used to get a slice from the array. However, [] in Symbol is used to get an output from a grouped symbol.

ChaiBapchya · 2018-12-16T22:13:18Z

docs/tutorials/gluon/hybrid.md

+    return x[0]
+```
+
+The current workaround is explicitly call [`slice`](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.slice) operators in `hybrid_forwar`d.


nit: to explicitly call

ChaiBapchya · 2018-12-16T22:13:48Z

docs/tutorials/gluon/hybrid.md

+
+#### In-Place Arithmetic Operators
+
+In place arithmetic operators are also used a lot in Gluon imperative mode. You need to avoid that and write the operations explicitly in `hybrid_forward`.


nit : In-place

ChaiBapchya · 2018-12-16T22:14:38Z

docs/tutorials/gluon/hybrid.md

+#### In-Place Arithmetic Operators
+
+In place arithmetic operators are also used a lot in Gluon imperative mode. You need to avoid that and write the operations explicitly in `hybrid_forward`.
+For example, avoid `x += y` and use `x  = x + y`, other wise you will get `NotImplementedError`.


nit: otherwise

roywei · 2018-12-18T18:31:54Z

@aaronmarkham @zheng-da could you help review? Thanks!

aaronmarkham

Made some suggested changes.

Line 3 mentions a newer version. That will deflect most traffic to this page, yet you're updating it with a lot of info. I'd Change that line to read:

## Related Content
* [Fast, portable neural networks with Gluon HybridBlocks](https://gluon.mxnet.io/chapter07_distributed-learning/hybridize.html)

Also, to clean up the formatting, why not try to use explicit math output with the math directive.

:math: `x.iadd(y)` <=> `x+=y`

aaronmarkham · 2018-12-18T19:12:27Z

docs/tutorials/gluon/hybrid.md

@@ -137,4 +137,94 @@ to gluon with `SymbolBlock`:
 net2 = gluon.SymbolBlock.imports('model-symbol.json', ['data'], 'model-0001.params')
 ```

+## Operators that does not work with hybridize


Suggested change

## Operators that does not work with hybridize

## Operators that do not work with hybridize

aaronmarkham · 2018-12-18T19:15:50Z

docs/tutorials/gluon/hybrid.md

@@ -137,4 +137,94 @@ to gluon with `SymbolBlock`:
 net2 = gluon.SymbolBlock.imports('model-symbol.json', ['data'], 'model-0001.params')
 ```

+## Operators that does not work with hybridize
+
+If you want to hybridize your model, you must use `F.some_operator` in your 'hybrid_forward' function, F will be 'mxnet.nd' before you hybridize and 'mxnet.sym' after hybridize. While most APIs are the same in NDArray and Symbol,  there are some differences. Writing `F.some_operator` and call `hybridize` may not work all the time.


Suggested change

If you want to hybridize your model, you must use `F.some_operator` in your 'hybrid_forward' function, F will be 'mxnet.nd' before you hybridize and 'mxnet.sym' after hybridize. While most APIs are the same in NDArray and Symbol, there are some differences. Writing `F.some_operator` and call `hybridize` may not work all the time.

If you want to hybridize your model, you must use `F.some_operator` in your 'hybrid_forward' function.

`F` will be 'mxnet.nd' before you hybridize and 'mxnet.sym' after hybridize. While most APIs are the same in NDArray and Symbol, there are some differences. Writing `F.some_operator` and call `hybridize` may not work all of the time.

aaronmarkham · 2018-12-18T19:16:36Z

docs/tutorials/gluon/hybrid.md

+### Element-wise Operators
+
+The following arithmetic and comparison APIs are automatically broadcasted if the input NDArrays have different shapes.
+However, that's not the case in Symbol API, it's not automatically broadcasted and you have to manually specify whether to use element-wise operator or broadcast operators.


Suggested change

However, that's not the case in Symbol API, it's not automatically broadcasted and you have to manually specify whether to use element-wise operator or broadcast operators.

However, that's not the case in Symbol API. It's not automatically broadcasted, and you have to manually specify whether to use element-wise operator or broadcast operators.

docs/tutorials/gluon/hybrid.md

aaronmarkham · 2018-12-18T19:23:37Z

docs/tutorials/gluon/hybrid.md

+
+### Slice
+`[]` in NDArray is used to get a slice from the array. However, `[]` in Symbol is used to get an output from a grouped symbol.
+For example, you will get different result for the following method before and after hybridization.


Suggested change

For example, you will get different result for the following method before and after hybridization.

For example, you will get different results for the following method before and after hybridization.

aaronmarkham · 2018-12-18T19:26:29Z

docs/tutorials/gluon/hybrid.md

+
+### Not implemented operators
+
+Some of the often used operators in NDArray  are not implemented in Symbol, and will cause hybridization failure


Suggested change

Some of the often used operators in NDArray are not implemented in Symbol, and will cause hybridization failure

Some of the often used operators in NDArray are not implemented in Symbol, and will cause hybridization failure.

aaronmarkham · 2018-12-18T19:30:01Z

docs/tutorials/gluon/hybrid.md

+
+#### Array API
+
+`mx.nd.array()` is used a lot but Symbol does not have the array API. The current workaround is to use F.ones/ F.zeros/ F.full which exists in both NDArrays and Symbols.


Suggested change

`mx.nd.array()` is used a lot but Symbol does not have the array API. The current workaround is to use F.ones/ F.zeros/ F.full which exists in both NDArrays and Symbols.

`mx.nd.array()` is used a lot, but Symbol does not have the `array` API. The current workaround is to use `F.ones`, `F.zeros`, or `F.full`, which exist in both the NDArray and Symbol APIs.

aaronmarkham · 2018-12-18T19:31:50Z

docs/tutorials/gluon/hybrid.md

+
+#### In-Place Arithmetic Operators
+
+In-place arithmetic operators are also used a lot in Gluon imperative mode. You need to avoid that and write the operations explicitly in `hybrid_forward`.


"avoid that"? What exactly? All in-place operators? Can you clarify this?
Maybe better to say something like...

Suggested change

In-place arithmetic operators are also used a lot in Gluon imperative mode. You need to avoid that and write the operations explicitly in `hybrid_forward`.

In-place arithmetic operators may be used in Gluon imperative mode, however if you expect to hybridize, you should write the operations explicitly instead.

avoid all in-place operators. I have added example: avoid x+=y and use x = x + y.

aaronmarkham · 2018-12-18T19:42:37Z

docs/tutorials/gluon/hybrid.md

+
+| NDArray in-place arithmetic operators | Description |
+|---|---|
+|[*NDArray.\__iadd\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__iadd__) |	x.__iadd__(y) <=> x+=y |


Notice that the double underscore is bolding the text. I'd be worried what this will be interpreted as by Sphinx/rst down the line.
You can escape it, or like I mentioned before use :math:.

roywei · 2018-12-19T20:21:07Z

@aaronmarkham @ChaiBapchya Thanks for the review, I have addressed your comments.

I have also added dive into deeplearning book as related content, let me know if it's not ready.
For the table format, I was following the format on website. As for the math formulas, it seems only renders on website, but not on downloaded jupyter notebooks. Let me knwo if you have other suggestions.
Added asnumpy and a summary section
Thanks!

roywei · 2018-12-21T17:54:33Z

@ThomasDelteil @nswamy Could you help take a look? Thanks!

aaronmarkham · 2018-12-21T22:49:28Z

docs/tutorials/gluon/hybrid.md

@@ -1,6 +1,9 @@
 # Hybrid - Faster training and easy deployment

-*Note: a newer version is available [here](http://gluon.mxnet.io/chapter07_distributed-learning/hybridize.html).*
+*Related Contents:*


nit: Content instead of Contents (but don't restart CI just for that change).

I added a link to the new dive into deeplearning book so made it "contents"

It's a semantic thing I guess. There's probably a rule here but... usually you can say Content is plural already (as it refers to a mass of stuff) and you don't usually say "Related Contents". It's more common to say "Related Content" even if there's more than one item in the list.
It is weird because you do say "Table of Contents", not "Table of Content".
🤷‍♂️ English is weird. It's not a showstopper... it's fine whichever way you want to do it. (but do it my way). Jk

aaronmarkham

Super minor nits.

aaronmarkham · 2018-12-21T22:53:10Z

docs/tutorials/gluon/hybrid.md

+
+### Not implemented operators
+
+Some of the often used operators in NDArray are not implemented in Symbol, and will cause hybridization failure


Suggested change

Some of the often used operators in NDArray are not implemented in Symbol, and will cause hybridization failure

Some of the often used operators in NDArray are not implemented in Symbol, and will cause hybridization failure.

aaronmarkham · 2018-12-21T22:53:34Z

docs/tutorials/gluon/hybrid.md

+Some of the often used operators in NDArray are not implemented in Symbol, and will cause hybridization failure
+
+#### NDArray.asnumpy
+Symbol does not support `asnumpy` function, you need to avoid calling `asnumpy` in `hybrid_forward`.


Suggested change

Symbol does not support `asnumpy` function, you need to avoid calling `asnumpy` in `hybrid_forward`.

Symbol does not support the `asnumpy` function. You need to avoid calling `asnumpy` in `hybrid_forward`.

aaronmarkham · 2018-12-21T22:54:41Z

docs/tutorials/gluon/hybrid.md

@@ -1,6 +1,9 @@
 # Hybrid - Faster training and easy deployment

-*Note: a newer version is available [here](http://gluon.mxnet.io/chapter07_distributed-learning/hybridize.html).*
+*Related Contents:*


Suggested change

*Related Contents:*

*Related Content:*

Roshrini · 2019-01-02T19:31:29Z

Seems like all the comments are addressed.
@sandeep-krishnamurthy Can you merge this PR?
@mxnet-label-bot Update [pr-awaiting-merge]

* update hybridize documentation * address review comments * improve doc * address comments * address comments

roywei requested a review from szha as a code owner December 10, 2018 04:57

piyushghai reviewed Dec 11, 2018

View reviewed changes

docs/tutorials/gluon/hybrid.md Outdated Show resolved Hide resolved

piyushghai reviewed Dec 11, 2018

View reviewed changes

docs/tutorials/gluon/hybrid.md Outdated Show resolved Hide resolved

marcoabreu added Doc pr-awaiting-review PR is waiting for code review labels Dec 12, 2018

ChaiBapchya reviewed Dec 16, 2018

View reviewed changes

roywei force-pushed the hybridize_doc branch from b25454a to ffd8e57 Compare December 18, 2018 00:12

roywei added 2 commits December 18, 2018 09:36

update hybridize documentation

e987fdd

address review comments

ee7df70

roywei force-pushed the hybridize_doc branch from ffd8e57 to ee7df70 Compare December 18, 2018 17:37

improve doc

ea44437

aaronmarkham suggested changes Dec 18, 2018

View reviewed changes

address comments

33cd247

aaronmarkham reviewed Dec 21, 2018

View reviewed changes

aaronmarkham approved these changes Dec 21, 2018

View reviewed changes

address comments

d2c8d0b

marcoabreu added pr-awaiting-merge Review and CI is complete. Ready to Merge and removed Doc pr-awaiting-review PR is waiting for code review labels Jan 2, 2019

aaronmarkham merged commit c63ef9a into apache:master Jan 7, 2019

thomelane mentioned this pull request Jan 8, 2019

a cheatsheet to document the issues in Gluon hybridize #12100

Closed

rondogency pushed a commit to rondogency/incubator-mxnet that referenced this pull request Jan 9, 2019

[MXNET-1255] update hybridize documentation (apache#13597)

c9538b3

* update hybridize documentation * address review comments * improve doc * address comments * address comments

haohuanw pushed a commit to haohuanw/incubator-mxnet that referenced this pull request Jun 23, 2019

[MXNET-1255] update hybridize documentation (apache#13597)

a2c5bdb

* update hybridize documentation * address review comments * improve doc * address comments * address comments

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MXNET-1255] update hybridize documentation #13597

[MXNET-1255] update hybridize documentation #13597

roywei commented Dec 10, 2018 •

edited

Loading

roywei commented Dec 12, 2018

ChaiBapchya left a comment

ChaiBapchya Dec 16, 2018

ChaiBapchya Dec 16, 2018

ChaiBapchya Dec 16, 2018

ChaiBapchya Dec 16, 2018

ChaiBapchya Dec 16, 2018

roywei commented Dec 18, 2018

aaronmarkham left a comment

aaronmarkham Dec 18, 2018

aaronmarkham Dec 18, 2018

aaronmarkham Dec 18, 2018

aaronmarkham Dec 18, 2018

aaronmarkham Dec 18, 2018

aaronmarkham Dec 18, 2018

aaronmarkham Dec 18, 2018

roywei Dec 19, 2018

aaronmarkham Dec 18, 2018

roywei commented Dec 19, 2018 •

edited

Loading

roywei commented Dec 21, 2018

aaronmarkham Dec 21, 2018

roywei Dec 21, 2018

aaronmarkham Dec 21, 2018

aaronmarkham left a comment

aaronmarkham Dec 21, 2018

aaronmarkham Dec 21, 2018

aaronmarkham Dec 21, 2018

Roshrini commented Jan 2, 2019 •

edited

Loading


		#### In-Place Arithmetic Operators

		In place arithmetic operators are also used a lot in Gluon imperative mode. You need to avoid that and write the operations explicitly in `hybrid_forward`.

	## Operators that does not work with hybridize
	## Operators that do not work with hybridize

	If you want to hybridize your model, you must use `F.some_operator` in your 'hybrid_forward' function, F will be 'mxnet.nd' before you hybridize and 'mxnet.sym' after hybridize. While most APIs are the same in NDArray and Symbol, there are some differences. Writing `F.some_operator` and call `hybridize` may not work all the time.
	If you want to hybridize your model, you must use `F.some_operator` in your 'hybrid_forward' function.
	`F` will be 'mxnet.nd' before you hybridize and 'mxnet.sym' after hybridize. While most APIs are the same in NDArray and Symbol, there are some differences. Writing `F.some_operator` and call `hybridize` may not work all of the time.

	However, that's not the case in Symbol API, it's not automatically broadcasted and you have to manually specify whether to use element-wise operator or broadcast operators.
	However, that's not the case in Symbol API. It's not automatically broadcasted, and you have to manually specify whether to use element-wise operator or broadcast operators.

	For example, you will get different result for the following method before and after hybridization.
	For example, you will get different results for the following method before and after hybridization.


		### Not implemented operators

		Some of the often used operators in NDArray are not implemented in Symbol, and will cause hybridization failure


		#### Array API

		`mx.nd.array()` is used a lot but Symbol does not have the array API. The current workaround is to use F.ones/ F.zeros/ F.full which exists in both NDArrays and Symbols.

	`mx.nd.array()` is used a lot but Symbol does not have the array API. The current workaround is to use F.ones/ F.zeros/ F.full which exists in both NDArrays and Symbols.
	`mx.nd.array()` is used a lot, but Symbol does not have the `array` API. The current workaround is to use `F.ones`, `F.zeros`, or `F.full`, which exist in both the NDArray and Symbol APIs.


		#### In-Place Arithmetic Operators

		In-place arithmetic operators are also used a lot in Gluon imperative mode. You need to avoid that and write the operations explicitly in `hybrid_forward`.

	In-place arithmetic operators are also used a lot in Gluon imperative mode. You need to avoid that and write the operations explicitly in `hybrid_forward`.
	In-place arithmetic operators may be used in Gluon imperative mode, however if you expect to hybridize, you should write the operations explicitly instead.

	Symbol does not support `asnumpy` function, you need to avoid calling `asnumpy` in `hybrid_forward`.
	Symbol does not support the `asnumpy` function. You need to avoid calling `asnumpy` in `hybrid_forward`.

[MXNET-1255] update hybridize documentation #13597

[MXNET-1255] update hybridize documentation #13597

Conversation

roywei commented Dec 10, 2018 • edited Loading

Description

Checklist

Essentials

Changes

Comments

roywei commented Dec 12, 2018

ChaiBapchya left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roywei commented Dec 18, 2018

aaronmarkham left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roywei commented Dec 19, 2018 • edited Loading

roywei commented Dec 21, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaronmarkham left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Roshrini commented Jan 2, 2019 • edited Loading

roywei commented Dec 10, 2018 •

edited

Loading

roywei commented Dec 19, 2018 •

edited

Loading

Roshrini commented Jan 2, 2019 •

edited

Loading