RNN

• Language modeling: predicting what word comes next
• LM is a benchmark task that helps us measure our progress or predicting language use
  • Pre-DL: n-grams
• The standard evaluation matrix is perplexity

$$\text{Perplexity}={\left(\prod _{t=1}^T\frac{1}{P_{LM}(x_{t+1}\mid x_1,\dots ,x_t)}\right)}^{\frac{1}{T}}$$

The core concepts here are:

1. word embedding with fixed window
2. we make a hidden layer
3. we softmax the distribution

RNN are a family of neural architectures: everytime you insert a new hidden state, you use multiple layers.

We multiply probability of a precedent state and we use it for the following states.

$$J(\Theta )=\frac{1}{T}\sum _{t=1}^T{J}^{(t)}(\Theta )$$

This is the sum of all losses.

The gradient wrt (W_h) is given by summing over all timesteps (or relevant cells):

$$\frac{\partial J}{\partial W_h}=\sum _{t=1}^T\frac{\partial J^{(t)}}{\partial W_h}$$

This is also called Backpropagation over time. BPTT.

Backpropagation.