Machine Learning · Insper · E2.3

Word Embeddings

Advanced approach

Other roads to word vectors, and how far their geometry can be trusted

The path of this part

  1. Other roadsLSA, GloVe, fastText
  2. Analogies under the microscopethe exclusion rule; offsets measured
  3. Biasa gender direction and its caveats
Each stage answers one question: which other roads lead to dense word vectors? → how much of an analogy is geometry? → what did the vectors learn about people?
Stage 1 · which other roads lead to dense word vectors?

The same compression on documents: LSA

Cosine of word pairs before and after the SVD of the TF-IDF matrix: car/automobile 0.07 to 0.64, gun/firearm 0.15 to 0.92, unrelated pairs stay at 0.00–0.01

Latent semantic analysis (LSA, Deerwester et al., 1990): truncated SVD (singular value decomposition) of the TF-IDF matrix (term frequency × inverse document frequency), 11,314 posts, 100 components.

car/automobile 0.07 → 0.64; gun/firearm 0.15 → 0.92; unrelated pairs stay at 0.00–0.01.

A whole post as context gives words of the same topic (car: nissan, mileage). A small window gives words of the same kind (car: truck, vehicle).

GloVe and fastText

GloVe (global vectors, Pennington et al., 2014)

Back to counting: fit vectors whose dot products approximate the log of the co-occurrence counts of a huge corpus.

fastText (Bojanowski et al., 2017)

A word is the sum of its character n-grams:

latiu → <la, lat, ati, tiu, iu>, <lat, lati, …

6 pieces shared with latido

A word never seen still gets a vector, and inflected forms (latir, latiu, latia) end up close.
Stage 2 · how much of an analogy is geometry?

The quiet trick: the input words are excluded

Predict. most_similar never returns an input word. Without that rule, what is nearest to king − man + woman?

nothing excludednearest three
king − man + womanking 0.89, queen 0.85, throne 0.77
doctor − man + womandoctor 0.90, nurse 0.84, child 0.77
  • "doctor − man + woman = nurse" appears only because "doctor" is forbidden (Nissim et al., 2020).
  • cos(king − queen, man − woman) = 0.58; an unrelated offset: −0.05.
Analogies are a real but approximate regularity.
Stage 3 · what did the vectors learn about people?

Bias: what the vectors learned about people

Average 8 differences (he − she, man − woman, …) into a gender direction; project occupations on it.

receptionist−0.31
nurse−0.30
teacher0.00
engineer0.24
manager0.37
captain0.42

For reference: she −0.19, he 0.24. Receptionist leans further to "she" than "she"; captain further to "he" than "he".

The vectors compressed who was written about in which role. A system built on them inherits the lean; removing it is an open problem.

Reading a bias table with care

the direction is estimated

From 8 pairs of words. One pair alone can be noisy: "man" projects to only +0.10.

a word moves for other reasons

In news, "captain" is often a sports or military title, not only a gendered role.

  • A system that ranks résumés by similarity to a job description inherits the lean.
  • Projecting the direction out (Bolukbasi et al., 2016) hides the bias more than it removes it.
Measuring is the first step; removing it is an open problem.

Summary

  1. A whole document as the context (LSA) gives neighbors of the same topic; a small window gives neighbors of the same kind.
  2. GloVe fits the log of global co-occurrence counts; fastText builds a word from its character pieces, so unseen words get vectors, and spelling neighbors creep in.
  3. most_similar excludes the input words: much of the famous analogy accuracy comes from forbidding the obvious wrong answer.
  4. A gender direction exposes stereotypes the vectors compressed from their text; read it with its caveats.
Advanced lab · about 25 min, in pairs

Scoring the vectors, steps 6 to 8

  • Same notebook, same session: the steps use vocab, pos, chunks, W_svd, W_w2v, W_glove and glove.
  • The analogy test (3,732 questions), fastText on words never seen, and a map of the space with PCA (principal component analysis) and t-SNE (t-distributed stochastic neighbor embedding).
  • Simulations: the exclusion rule at scale (step 6) and does the map keep the neighbors? (step 8).

Pace: step 6 and the first simulation in 10 min; step 7 in 6 min; step 8 and the second in 9 min.