How to quantize document

docs/index.rst

@@ -16,3 +16,4 @@ Blueoil -- An efficient neural network generator
    faq/index
    reference/index
    converter/index
+   quantize/index

docs/quantize/activation_quantization.rst

@@ -0,0 +1,48 @@
+Activation quantization
+=======================
+
+Blueoil can quantize an activation function by passing the callable ``activation quantizer`` and keyword arguments ``activation_quantizer_kwargs`` to the network class.
+
+.. literalinclude:: ../../blueoil/networks/classification/quantize_example.py
+   :language: python
+   :lines: 46-59, 70-79
+   :emphasize-lines: 8-9, 21,24
+
+Activation quantizer
+--------------------
+
+Currenly, Blueoil has only one activation function quantizer.
+
+Linear mid tread half quantizer (``LinearMidTreadHalfQuantizer``)
+_________________________________________________________________
+
+This quantization creates a linear mid tread half quantizer.
+If ``backward`` is provided, this ``backward`` will be used in backpropagation.
+
+This quantization method is DoReFa-Net [1]_ activation quantization variant, the differencce from DoReFa-Net is to be able to change ``max_value``.
+
+Forward is:
+
+.. math::
+    \mathbf{X} & = \text{clip}\big(\mathbf{X}, 0, max\_value\big)\\
+    \mathbf{Y} & =
+        \begin{cases}
+        \mathbf{X},  & \text{if $bit$ is 32} \\
+        \frac{\text{round}\big(\frac{\mathbf{X}}{max\_value}
+            \cdot (2^{bit}-1)\big)}{2^{bit}-1} \cdot max\_value, & otherwise
+        \end{cases}
+
+Default backward is:
+
+.. math::
+    \frac{\partial Loss}{\partial \mathbf{X}} =
+        \begin{cases}
+        \frac{\partial Loss}{\partial y},  & \text{if $0 < x < max\_value$}\\
+        0, & otherwise
+        \end{cases}
+
+
+Reference
+
+    - `Deep Learning with Low Precision by Half-wave Gaussian Quantization <https://arxiv.org/abs/1702.00953>`_
+    .. [1] `DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients <https://arxiv.org/abs/1606.06160>`_

docs/quantize/index.rst

@@ -0,0 +1,17 @@
+.. Converter documentation master file, created by
+   sphinx-quickstart on Mon Apr  9 12:52:21 2018.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Quantize Documentation
+======================
+
+Welcome to Blueoil Quantize documentation.
+Quantize is a quantization part of Blueoil for edge devices.
+
+.. toctree::
+   :maxdepth: 1
+
+   weight_quantization
+   activation_quantization
+   quantization_template

docs/quantize/quantization_template.rst

@@ -0,0 +1,13 @@
+Simple Quantized network
+========================
+
+Quantize network example code
+-----------------------------
+
+.. literalinclude:: ../../blueoil/networks/classification/quantize_example.py
+   :language: python
+
+Quantize Network graph structure
+--------------------------------
+
+.. image:: ../_static/quantize_example_minimal_graph_with_shape.png

docs/quantize/weight_quantization.rst

@@ -0,0 +1,83 @@
+Weight quantization
+===================
+
+Blueoil can quantize weight in the network by passing the callable ``weight quantizer`` and keyword arguments ``weight_quantizer_kwargs`` to the network class.
+
+.. literalinclude:: ../../blueoil/networks/classification/quantize_example.py
+   :language: python
+   :lines: 46-59, 70-79
+   :emphasize-lines: 10-11, 20,23
+
+Tensorflow custom getter
+------------------------
+
+The main idea of selecting variable to do weight quantize in Blueoil is from the ``custom_getter`` in ``variable_scope`` namely ``_quantized_variable_getter``.
+
+.. literalinclude:: ../../blueoil/networks/classification/quantize_example.py
+   :language: python
+   :lines: 82-89, 103-
+   :emphasize-lines: 1, 11-25, 30-36
+
+The selection criteria is based on these three variables.
+
+    - ``name``: This is an argument for ``tf.compat.v1.variable_scope(name)`` which indicate the raw_ops in this layer.
+    - ``quantize_first_convolution``: boolean indicate quantization on the **first** convolution layer
+    - ``quantize_last_convolution``: boolean indicate quantization on the **last** convolution layer
+
+The variable which variable scope name ending with ``kernel`` will be weight quantized, except it is a first or last layer with ``quantize_first_convolution`` or ``quantize_last_convolution`` set as `False` respectively.
+
+Weight quantizer
+----------------
+
+Selection of weight quantizer are ``Binary channel wise mean scaling quantizer`` and ``Binary mean scaling quantizer``:
+
+Binary channel wise mean scaling quantizer (``BinaryChannelWiseMeanScalingQuantizer``)
+______________________________________________________________________________________
+
+This quantization creates a binary channel wise mean scaling quantizer.
+If ``backward`` is provided, this ``backward`` will be used in backpropagation.
+
+This method is varient of XNOR-Net [1]_ weight quantization, the differencce from XNOR-Net is backward function.
+
+Forward is:
+
+.. math::
+    \begin{align}
+        \bar{\mathbf{x}} & = \frac{1}{n}||\mathbf{X}||_{\ell1}
+        & \text{$\bar{\mathbf{x}}$ is a $c$-channels vector} \\
+        & & \text{$n$ is number of elements in each channel of $\mathbf{X}$} \\\\
+        \mathbf{Y} & = \text{sign}\big(\mathbf{X}\big) \times \bar{\mathbf{x}} &\\
+    \end{align}
+
+Default backward is:
+
+.. math::
+    \frac{\partial Loss}{\partial \mathbf{X}} = \frac{\partial Loss}{\partial \mathbf{Y}}
+
+Binary mean scaling quantizer (``BinaryMeanScalingQuantizer``)
+______________________________________________________________
+
+This quantization creates a binary mean scaling quantizer.
+If ``backward`` is provided, this ``backward`` will be used in backpropagation.
+
+This method is DoReFa-Net [2]_ weight quantization.
+
+Forward is:
+
+.. math::
+    \begin{align}
+        \bar{x} & = \frac{1}{N}||\mathbf{X}||_{\ell1}
+        & \text{$\bar{x}$ is a scalar} \\
+        & & \text{$N$ is number of elements in all channels of $\mathbf{X}$}\\
+        \mathbf{Y} & = \text{sign}\big(\mathbf{X}\big) \cdot \bar{x} &\\
+    \end{align}
+
+Default backward is:
+
+.. math::
+    \frac{\partial Loss}{\partial \mathbf{X}} = \frac{\partial Loss}{\partial \mathbf{Y}}
+
+Reference
+
+    .. [1] `XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks <https://arxiv.org/abs/1603.05279>`_
+    .. [2] `DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients <https://arxiv.org/abs/1606.06160>`_