Word2Vec

Word2Vec is a framework for learning word vectors, an alternative representation.

1. Corpus, a long list of words
2. Every word in a fixed vocabulary is represented by a vector
3. Go through each position in $t$ in the text which has a center word $c$ and outside context words
4. Use the similarity of the word vectors for $c$ and $o$ to calculate $P(o|c)$ and $P(c|o)$
5. Keep adjusting the word vectors to maximize probability

With a slidable window = 2:

  problems     turning   **into**    banking      crises
        outside           center           outside         
f(w_t-2|wt) f(w_t-1|wt)      c      f(w_t+1|wt)   f(w_t+2|wt)

So:

$$L(\varphi )=\prod _{t=1}^T\prod _{\begin{array}{c}-c\le j\le c\\ j\ne 0\end{array}}P(w_{t+j}\mid w_t;\varphi )$$

This is the product of each conditional probability for all words close to the center.

This is complessive, therefore works for each word related to center.

But we also need to calculate the loss function!

$$J(\theta )=-\sum _{t=1}^T\sum _{\begin{array}{c}-c\le j\le c\\ j\ne 0\end{array}}\log P(w_{t+j}\mid w_t;\theta )$$

Minimize = loss function = Maximum accuracy!

To get a single couple of word, C as central word and O as context word:

$$P(O\mid C)=\frac{\exp (U_o^{\top }{V}_c)}{{\sum }_{w\in V}\exp (U_w^{\top }{V}_c)}$$

This is the basic “unit” of predicting things, unlike $L(\varphi )$ that works for all the dataset.

This is also known as softmax.

We now need to calculate gradient: how much vectors need to be in order to reduce errors.