hw2.org

HW2-REPORT

MLQP Analyse

Forward

$$ x_ik = fk(nk_i) $$ $$ nk_i = bk_i + ∑j=1N^{k-1}(uki,j (xk-1_j)^2 + vki,j xk-1_j )$$

Backpropagation

Let:

$$ s^k_i = \frac{∂ F}{∂ n^k_i}$$

Then:

$$ u^ki,j(l+1) = u^ki,j(l) - α s^k_i \frac{∂ n^k_i}{∂ u^ki,j} = u^ki,j(l) - α s^k_i (xk-1_j)^2$$ $$ v^ki,j(l+1) = v^ki,j(l) - α s^k_i \frac{∂ n^k_i}{∂ v^ki,j} = v^ki,j(l) - α s^k_i xk-1_j$$ $$ b^ki,j(l+1) = b^ki,j(l) - α s^k_i \frac{∂ n^k_i}{∂ b^ki,j} = b^ki,j(l) - α s^k_i $$

Now calculate $s^k$ :

$$ s^k = \frac{∂ F}{∂ n^k} = \frac{∂ F}{∂ nk+1} \frac{∂ nk+1}{∂ n^k} = sk+1 \frac{∂ nk+1}{∂ n^k}$$

To calculate the Jacobbi’s matrix

$$ J^k = \frac{∂ nk+1}{∂ n^k} $$

We have:

$$ J^ki,j = \frac{∂ nk+1_i}{∂ n^k_j} = \frac{∂( bk+1_i + ∑l=1N^{k}(uk+1i,l (xk_l)^2 + vk+1i,l xk_l) )}{∂ n^k_j} = (2 x^k_j uk+1i,j + vk+1i,j) \frac{∂ x^k_j}{∂ n^k_j} $$

At last, calculate s^m :

$$ s^m_i = \frac{∂ ∑N^mj=1 (t_j-x^m_j)^2}{∂ n^m_i} = 2(x^m_i-t_i) \frac{∂ x^m_i}{∂ n^m_i}$$

Let

\begin{equation} \dot{F^k}_i,j = \begin{cases} \dot{f^k}(x^k_i), & i=j \cr 0, & i ≠ j \end{cases}\nonumber \end{equation}

\begin{equation} X^k_i,j = \begin{cases} x^k_i, & i=j \cr 0, & i ≠ j \end{cases}\nonumber \end{equation}

Then $$s^m = 2(x^m-t)\dot{F^m}$$ $$s^k = s^k+1J^k= s^k+1(\dot{F^k}V^k+1 + 2\dot{F^k} X^k U^k+1 )$$

Two-spirals problem

Problem

The two-spirals problem is defined as follow:

file:imgs/two-spirals.png

It is extremely hard for traditional network to regress such a pattern.

In the following I will try to use MLQP to solve this problem.

Net Structure

I try to use a two-layer MLQP network to solve the problem.The first layer contains 10 neurons with sigmoid function as transport function.The second layer contains 1 neuron with purelin function as transport function.

To avoid the weight matrixes always being symmetry,I gave each cell of them random initial values between -1 and 1.

As sigmoid function almost become a constant when x > 5, I use very small α s from 0.0001 to 0.001.

Result

The trainning result when α = 0.01:

file:imgs/result-0010.png

The trainning result when α = 0.005:

file:imgs/result-0005.png

The trainning result when α = 0.001:

file:imgs/result-0001.png

Each result use a random net and looped for 1000 times