LSTM builder is not the standard LSTM

[pmichel31415]

After looking at the code in lstm.h, it seems that the LSTMBuilder cell has the following update rule :

Which is not the common update rule :

(i.e. in the dynet implementation the forget and input gate are coupled and there are peaky connections with the cell state c)

What are the reasons behind this choice?

Anyway I think this should be documented.

[neubig]

I've heard that the motivation is the results in this paper: https://arxiv.org/abs/1503.04069

But I agree that this should be documented.

[pmichel31415]

I see. I started documenting the lstm.

I also started implementing a vanilla LSTM. Would it be interesting to have it in the form of two optional parameters coupling and peepholes in the constructor? With default values true for compatibility.

[yoavg]

the LSTMBuilder is quite central. I agree that it would be good to add
these, just need to ensure that:

• no speed penalty
• the values are saved / loaded with the model
• we remain backward compatible with saved models

On Tue, Nov 8, 2016 at 9:31 PM, Paul Michel [email protected]
wrote:

I see. I started documenting the lstm.

I also started implementing a vanilla LSTM. Would it be interesting to
have it in the form of two optional parameters coupling and peepholes in
the constructor? With default values true for compatibility.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#154 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAcVtHrC5LRkHsWELhZuViolpdL86VyJks5q8M4VgaJpZM4Kso4N
.

[yoavg]

After staring at the code (admittedly without pen and paper at hand, I'm commuting), I am pretty sure the code implements something else than what's written above -- for example $o_t$ uses $c_t$ (ct[i]) and not $c_{t-1}$ (i_c_tm1), and tanh is applied twice (it is also applied on $c_t$ just prior to computing $h_t$.

@redpony ?

[pmichel31415]

@yoavg yes you are right this is my mistake, I'll update the original post.

I've been working on implementing the aforementioned options to the LSTMBuilder, but I'll put it on hold until the dropout PR is merged.

[yoavg]

this may affect the dropout PR, as I may be introducing dropout in an un-intended place (or effectively introducing it twice) -- and may not be.

[neubig]

An alternative VanillaLSTMBuilder is now available, documentation in the "Unorthodox Things" section as of this commit: 7935e79