Vanishing Gradient Problem - LSTM

As far as we advance in RNN layers and we do Backpropagation, gradient becomes weaker and weaker.

To solve this: LSTM! Long-Short Term Memory, which work as computer’s RAM.

For each step t, there is a hidden state $h^t$ and a cell state $c^t$

• both are of the same $n$ length

• cell stores long-term information

• LSTM can insert, delete or read information from cell state

• The selection of keeping information or no is managed from three gates of n length, that can be 0, 1 or 0/1

  • Forget gate: $f^t=\sigma (W_f{h}^{t-1}+U_f{x}^t+b_f)$
  • Input gate: $i^t=\sigma (W_i{h}^{t-1}+U_i{x}^t+b_i)$
  • Output gate: $o^t=\sigma (W_o{h}^{t-1}+U_o{x}^t+b_o)$

${\tilde{c}}^t=\tanh (W_c{h}^{t-1}+U_c{x}^t+b_c)$

$c^t=f^t\odot c^{t-1}+i^t\odot {\tilde{c}}^t$

$h^t=o^t\odot \tanh (c^t)$