Skipgram model with negative samples

1. We take k samples
2. maximize probability the real outside words appear
3. minimize probability that random words appear around center word

$$J_{neg}-sample(u_o,v_c,U)=-log(\sigma )(W_0^T{V}_c)-\sum _{k\in \{ksampledindices\}}log\sigma (-U_k^T{V}_c)$$ $$\sigma (x)=\frac{1}{1+e^{-x}}$$

If we do a matrix to count all occurrences on a corpus, it would take forever.

So, the idea is to save important information somewhere.

We can use GLOVE: we can save frequency of each couple of words in a corpus

$$w_i{w}_j=log(P(i|j))$$