Word Embeddings

The problem: every word is its own column

• In bag-of-words and TF-IDF the columns are orthogonal.
• "car" and "automobile" are exactly as unrelated as "car" and "banana".
• The representation knows which words a document contains, nothing about what they mean.

The path of this lesson

A word you have never seen

• Um copo de tiquira gelada. (A glass of cold tiquira.)
• A tiquira é feita de mandioca fermentada. (… made from fermented cassava.)
• Bebeu tiquira demais e passou mal. (He drank too much … and felt sick.)
• No Maranhão, servem tiquira nas festas. (… they serve it at parties.)

The distributional hypothesis

You inferred it from the company it keeps: glass, cold, fermented, drank, too much, parties.

A toy corpus: 11 sentences

Designed so that gato/cachorro and carro/automóvel never share a sentence, but share their neighbors.

For each word, count every other word within a window of two words to the left or right: a 20 × 20 matrix, 126 (word, context) pairs.

One row by hand: "gato"

• "o gato bebe leite": the window holds o, bebe, leite.
• "o gato come peixe": the window holds o, come, peixe.

Documents versus contexts

PMI: counting what is surprising

Not "how often together?" but "how much more often than chance?" (Church & Hanks, 1990)

o is next to everything (22 times as a context): barely surprising. The order of the raw counts flips. Clip negatives to zero: PPMI \(= \max(\text{PMI}, 0)\).

PPMI fixes the article, and becomes too literal

pair	counts	PPMI
gato / cachorro	0.73	0.22
carro / automóvel	0.87	0.75
gato / carro	0.50	0.05

Cat and dog share only bebe and come: milk, water, fish, meat are four orthogonal columns.

Truncated SVD: 20 sparse columns → 2 dense numbers

The whole section in four columns

The same compression on documents: LSA

Latent semantic analysis (LSA, Deerwester et al., 1990): truncated SVD of the TF-IDF matrix, 11,314 posts, 100 components.

car/automobile 0.07 → 0.64; gun/firearm 0.15 → 0.92; unrelated pairs stay at 0.00–0.01.

A whole post as context gives words of the same topic (car: nissan, mileage). A small window gives words of the same kind (car: truck, vehicle).

word2vec: learning by prediction

Mikolov et al., 2013. Slide a window and make training examples:

• skip-gram: the center predicts each neighbor → 18 examples from 6 words;
• CBOW: the averaged neighbors predict the center → 6 examples.

How the vectors drift together

• Every word starts with a random vector. A pair (center, context) is scored by the dot product.
• Training nudges the vectors so that pairs that really occur score high.
• Words in the same contexts get the same nudges: the distributional hypothesis as an optimization.

Training it: the knobs of gensim

GloVe and fastText

Pretrained vectors: GloVe, 400,000 words × 50 numbers

No single number means anything ("car" starts with 0.48, −0.08, 1.46…); only the geometry does. "python" mixes the snake, Monty Python and programming.

Similar is not synonymous

Analogies, measured honestly

\(\text{king} - \text{man} + \text{woman} \approx\ ?\)

Capitals, tenses, comparatives and languages mostly work; rare relations are not straight directions.

The quiet trick: the input words are excluded

• "doctor − man + woman = nurse" appears only because "doctor" is forbidden (Nissim et al., 2020).
• cos(king − queen, man − woman) = 0.58; an unrelated offset: −0.05.

Simulator: seeing the space

Predict. Click a word and compare its neighbors in 50-D (the lines) with the map: does the PCA map (24.5% of the variance) lie? Then untick "exclude".

Bias: what the vectors learned about people

Average 8 differences (he − she, man − woman, …) into a gender direction; project occupations on it.

The limit: one vector per word

• A vector is roughly a frequency-weighted average of the senses (Arora et al., 2018). Portuguese: banco, manga, vela.
• "The dog bit the man" / "the man bit the dog": same words, so word vectors alone cannot tell who bit whom.

Summary

1. Distributional hypothesis: similar contexts → similar vectors.
2. By hand: co-occurrence counts → PPMI (remove the obvious) → SVD (generalize).
3. word2vec predicts neighbors; it lands near the same place as counting. fastText uses pieces of words.
4. Pretrained vectors: neighbors, relatedness (antonyms too), approximate analogies, absorbed bias.
5. One vector per word, whatever the sense.

Word vectors three ways

• The first million words of text8 (English Wikipedia, cleaned).
• Counting (PPMI + SVD), prediction (word2vec) and pretrained (GloVe): neighbors, the analogy test (3,732 questions), fastText on words never seen, a map of the space.
• Download the lab notebook and open it in Jupyter or Colab; in Colab run !pip install gensim first.
• Run the blocks in order and compare with the outputs on the page.

Downloads: 31 MB (text8) and 66 MB (GloVe).