Gradient

$J(\theta )$ is the loss function; its gradient is $\nabla J(\theta )$. The algorithm is called gradient descent.

It points in the direction of the steepest increase of the function, so moving in the opposite direction decreases the loss fastest.

The idea:

• for the current value of $\varphi$, calculate the gradient of $J(\varphi )$
• then take a small step in the direction of the negative gradient.

$${\theta }^{\text{new}}={\theta }^{\text{old}}-\alpha \nabla J({\theta }^{\text{old}})$$

Therefore:

$${\theta }_j^{new}={\theta }_j^{old}-\alpha \frac{\partial }{\partial {\theta }_j^{old}}J(\theta )$$

The problem is that $J(\varphi )$ works for all the corpus but we cannot wait for termination for every computation.

We can use SGD:

• SGD: skipgrams $\to$ predict outside given center
• CBOW: center bag of words $\to$ predict center from outside words