Machine Learning · Insper · E2.2

TF-IDF

Advanced approach

Term frequency × inverse document frequency: the knobs of the vectorizer, and where every sparse vector breaks

The path of this part

Turn the knobssmooth_idf, sublinear_tf, norm
What it cannot seesynonyms, word order, two meanings
Words it never sawno column, a zero vector

Each stage answers one question: what do the knobs change? → what can sparse vectors not see? → what happens to words never seen?

Stage 1 · what do the knobs change?

The defaults, and three knobs

\[ \text{tf-idf}(t, d) = \text{tf}(t, d) \times \Big(\ln \frac{1 + n}{1 + \text{df}(t)} + 1\Big) \]

\(n\) documents, \(\text{df}(t)\) of them contain the word \(t\), \(\text{tf}(t, d)\) is its count in document \(d\); then each row is divided by its length.

knob	default	the other choice
`smooth_idf`	`True`: the "+1" inside the fraction	`False`: \(\ln(n/\text{df}) + 1\)
`sublinear_tf`	`False`: tf is the raw count	`True`: \(1 + \ln(\text{tf})\)
`norm`	`'l2'`: each row of length 1	`None`: the raw products

Predict. Row d3 = "Gato dorme, gato sonha." has gato twice (idf 1) and two rare words (idf 1.6931). Which knob could stop the two gato from outweighing the rare words?

Three knobs, on row d3

option	dorme	gato	sonha
default	0.5427	0.6411	0.5427
`smooth_idf=False`	0.5864	0.5588	0.5864
`sublinear_tf=True`	0.5774	0.5774	0.5774
`norm=None`	1.6931	2	1.6931

smooth_idf=False: \(\ln(n/\text{df}) + 1\); the rare words get \(\ln 3 + 1 = 2.0986\) and overtake "gato".
sublinear_tf=True: count → \(1 + \ln(\text{tf})\); the two "gato" count \(1.6931\), equal to the rare words' idf by coincidence.
norm=None: raw products; longer documents get larger vectors (matters for Euclidean distance, not for the cosine).

Stage 2 · what can sparse vectors not see?

The limits of sparse representations

Each word is its own axis, orthogonal to every other word.

Predict. "o carro é rápido" and "o automóvel é veloz" (the car is fast / the automobile is quick): what is their cosine?

pair	cosine	shared words
synonyms: o carro é rápido / o automóvel é veloz	0.00	none
word order: o cachorro mordeu o homem / o homem mordeu o cachorro	1.00	cachorro, homem, mordeu
two meanings: a manga da camisa rasgou / comi uma manga madura	0.14	manga
same meaning: a manga da camisa rasgou / o punho da blusa descosturou	0.14	da

Stage 3 · what happens to words never seen?

And a fifth: unseen words have no column


'o cachorro latiu'       non-zeros: 1   {'cachorro': 1.0}
'um felino adormeceu'    non-zeros: 0   {}

"a feline fell asleep": a sentence about a cat sleeping, next to documents about cats sleeping, with similarity 0 to all of them.

By Heaps' law, new words never stop arriving. These failures motivated the dense representations (E2.3 and E2.4).

Summary

smooth_idf, sublinear_tf and norm change the weights; sublinear_tf stops a repeated word from taking over its row.
Every word is its own axis: synonyms score 0, word order is invisible, one spelling with two meanings looks alike.
A word never seen in fit has no column; a text made only of new words is the zero vector.

These failures motivated the dense representations (E2.3 and E2.4).

Advanced lab · about 15 min, in pairs

A TF-IDF search engine on real posts, steps 6 and 7

Continue in the same notebook and session as steps 1 to 5.
Step 6: six queries written by hand, scored by how many of the 5 results come from the right group.
Step 7: "image" and "picture", "faith" and "belief": what keyword search cannot see.
Simulation: keep the rare words (step 6).
The lab notebook, from step 6.

Pace: step 6 and the simulation in 10 min; step 7 in 5 min.