Machine Learning · Insper · E2.3

Word Embeddings

Fundamental approach

How a computer learns what a word means from the words around it: by hand (counts → PPMI, positive pointwise mutual information → SVD, singular value decomposition), by prediction (word2vec) and pretrained (GloVe)

The problem: every word is its own column

In bag-of-words and TF-IDF the columns are orthogonal.
"car" and "automobile" are exactly as unrelated as "car" and "banana".
The representation knows which words a document contains, nothing about what they mean.

A word embedding replaces the column with a short, dense vector (50 to 300 numbers), arranged so that words used in similar ways get similar vectors.

The path of this part

Context → meaningthe distributional hypothesis
Count and compressco-occurrence counts → PPMI → truncated SVD
Learn by predictingskip-gram, CBOW, negative sampling
Explore the spaceneighbors, relatedness, analogies, one vector per word

Each stage answers one question: where does a word's meaning come from? → how do counts become a dense vector? → can a model learn the vectors by predicting? → what do the vectors capture, and what do they miss?

Stage 1 · where does a word's meaning come from?

A word you have never seen

Um copo de tiquira gelada. (A glass of cold tiquira.)
A tiquira é feita de mandioca fermentada. (… made from fermented cassava.)
Bebeu tiquira demais e passou mal. (He drank too much … and felt sick.)
No Maranhão, servem tiquira nas festas. (… they serve it at parties.)

Predict. Nobody defined it. What is tiquira, and how do you know?

The distributional hypothesis

You inferred it from the company it keeps: glass, cold, fermented, drank, too much, parties.

Harris, 1954

words that occur in similar contexts tend to have similar meanings

Firth, 1957

"you shall know a word by the company it keeps"

Meaning becomes something you can count: if two words keep the same neighbors, their vectors should be close.

Stage 2 · how do counts become a dense vector?

From counts to PPMI, step by step

Eleven sentences, a window of two words; PMI = pointwise mutual information. Predict. In raw counts, which context of "gato" is the strongest? After PMI, which one is the weakest?

Truncated SVD: 20 sparse columns → 2 dense numbers

word	dim 1	dim 2
gato	3.77	−0.97
cachorro	2.55	−0.35
leite	2.78	−0.79
carro	0.61	1.87
automóvel	0.70	2.28
gasolina	0.64	2.44

The SVD (PCA, principal component analysis, without centering) keeps the \(k\) directions that explain most of the matrix.

Animals and food: big first coordinate. Vehicles and fuel: big second.

Nobody told it which words are animals: the groups share contexts inside, not across.

The whole section in four columns

pair	counts	PPMI	PPMI + SVD
gato / cachorro	0.73	0.22	0.99
carro / automóvel	0.87	0.75	1.00
gato / carro	0.50	0.05	0.06

SVD of raw counts mixes animals and vehicles along one ray; SVD of PPMI puts animals and food in one corner and vehicles and fuel in another

Stage 3 · can a model learn the vectors by predicting?

word2vec: learning by prediction

A six-word sentence with a window of two around bebe: skip-gram makes four examples, CBOW one, negative sampling pairs the real example with random ones

Mikolov et al., 2013. Slide a window and make training examples:

skip-gram: the center predicts each neighbor → 18 examples from 6 words;
CBOW (continuous bag-of-words): the averaged neighbors predict the center → 6 examples.

How the vectors drift together

Every word starts with a random vector. A pair (center, context) is scored by the dot product.
Training nudges the vectors so that pairs that really occur score high.
Words in the same contexts get the same nudges: the distributional hypothesis as an optimization.

negative sampling

Instead of scoring all 100,000 words: "did these two really appear together?" Real (bebe, leite) → 1; random (bebe, diesel) → 0, with 5 to 20 random words per real pair.

Levy & Goldberg (2014): skip-gram with negative sampling factorizes a shifted PMI matrix. Prediction and counting reach the same place.

Training it: the knobs of gensim

parameter	what it does
`vector_size`	numbers per word (100 to 300 is typical)
`window`	small (2–5): same function; large (10+): same topic
`sg`	1 = skip-gram (slower, better for rare words); 0 = CBOW
`negative`	random words per real pair
`min_count`	rarer words are dropped
`epochs`, `seed`	passes; `seed` and `workers=1` for reproducibility

Stage 4 · what do the vectors capture, and what do they miss?

Pretrained vectors: GloVe, 400,000 words × 50 numbers

Predict. What are the five nearest words of "python"?

word	nearest by cosine
car	truck 0.92, cars 0.89, vehicle 0.88, driver 0.85, driving 0.84
mango	pineapple 0.91, guava 0.90, papaya 0.89, pear 0.85, coconut 0.85
brazil	portugal 0.88, argentina 0.87, rica 0.84, spain 0.83, costa 0.83
python	reticulated 0.69, spamalot 0.66, php 0.64, owl 0.63, mouse 0.63

GloVe (global vectors): trained at Stanford on 6 billion tokens. No single number means anything ("car" starts with 0.48, −0.08, 1.46…); only the geometry does. "python" mixes the snake, Monty Python and programming.

Similar is not synonymous

Predict. hot / cold: a high or a low cosine?

car / automobile	0.70
doctor / physician	0.80
hot / cold	0.80
good / bad	0.80
car / banana	0.22

An antonym is the most interchangeable word there is: "the soup is hot" / "the soup is cold".

The hypothesis measures relatedness, not synonymy: fine for search, a trap for sentiment.

Analogies, measured honestly

\(\text{king} - \text{man} + \text{woman} \approx\ ?\)

hits

queen · tokyo · rome · swam · faster · worse · spanish

7 / 12

misses

bigger − big + small = larger
brother − man + woman = daughter
glove − hand + foot = "50,000-square"

Capitals, tenses, comparatives and languages mostly work; rare relations are not straight directions.

Simulator: seeing the space

Predict. Click a word and compare its neighbors in 50-D (the lines) with the map: does the PCA map (24.5% of the variance) lie? Then try a preset analogy.

The limit: one vector per word

bank

banks, securities, banking, investment, exchange, financial…

bank/money 0.64 · bank/river 0.41

apple

blackberry, chips, iphone, microsoft, ipad…

the fruit is nowhere in the top ten

A vector is roughly a frequency-weighted average of the senses (Arora et al., 2018). Portuguese: banco, manga, vela.
"The dog bit the man" / "the man bit the dog": same words, so word vectors alone cannot tell who bit whom.

Next page, sentence embeddings: the vector of each word computed from the sentence around it.

Summary

Distributional hypothesis: similar contexts → similar vectors.
By hand: co-occurrence counts → PPMI (keep what beats chance) → SVD (merge contexts that behave alike).
word2vec predicts the neighbors; with negative sampling, yes/no questions. It lands near the same place as counting.
Pretrained vectors: neighbors measure relatedness (antonyms too); analogies work part of the time.
One vector per word, whatever the sense.

Advanced part: other roads (LSA, GloVe, fastText), analogies under the microscope, bias.

Fundamental lab · about 35 min, in pairs

Word vectors three ways, steps 1 to 5

The first million words of text8 (English Wikipedia, cleaned).
Counting (PPMI + SVD), prediction (word2vec) and pretrained (GloVe), compared by their neighbors on the same words.
Download the lab notebook and open it in Colab or Jupyter; in Colab run !pip install gensim first.
At every step, answer the Predict before running.
Simulations: what PPMI is for (step 3) and skip-gram or CBOW (step 4).

Pace: steps 1 and 2 in 8 min; step 3 and the first simulation in 10 min; step 4 and the second in 10 min; step 5 in 7 min.