Text Representation

The problem: text is not a table

• PCA (principal component analysis) rotates the columns of a table; k-means measures distances between its rows.
• A review, a support ticket, a forum post is a string of any length.

This lesson builds the bridge from strings to vectors, so that distances, projections and clusters apply to text.

Two families of answers

The path of this lesson

Cut the text into tokens

from sklearn.feature_extraction.text import CountVectorizer

docs = ["O gato comeu o peixe.",                 # the cat ate the fish
        "O cachorro comeu a ração do gato.",     # the dog ate the cat's food
        "Gato dorme, gato sonha."]               # cat sleeps, cat dreams
analyzer = CountVectorizer().build_analyzer()    # scikit-learn's default tokenizer
for d in docs:
    print(analyzer(d))
  

Three silent decisions

['gato', 'comeu', 'peixe']
['cachorro', 'comeu', 'ração', 'do', 'gato']
['gato', 'dorme', 'gato', 'sonha']
  

Three tokenizers, one sentence

s = "Os gatos não comeram a ração do cachorro, mas o gato comeu!" (the cats did not eat the dog's food, but the cat ate!)

Stop words and n-grams

The example sentence: "Os gatos não comeram a ração do cachorro, mas o gato comeu!" (the cats did not eat the dog's food, but the cat ate; 12 tokens)

On the three sentences of the previous slides: 8 columns → 7 with a tiny stop list → 17 with bigrams.

Stemming and lemmatization

Merging the variants of a word into one token. On the same sentence:

The same sentence, normalized

• "com" (with) and "comida" (food) also stem to "com": four unrelated words share one column.
• The small lemmatizer left "comeu" as it is: it is a statistical model, and it makes mistakes.
• "não" survives this stop list; scikit-learn's English list removes "not", "no", "never".

Subwords: the tokenizers of neural models

Frequent words stay whole, rare words split into frequent pieces: no word is ever unknown.

##: glued to the piece before; ▁: start of a word. Each piece has its own vector, so a word never seen whole still gets one.

Where each stage-1 decision applies

How big does the vocabulary get?

The 20 Newsgroups collection: 11,314 posts from Usenet groups of the early 1990s. Read them one token at a time and count the distinct words seen so far.

Heaps' law: the vocabulary never stops growing

how to read it

• x axis: tokens read so far; y axis: distinct words seen. Both on a log scale.
• On a log scale a power law is a straight line; its slope, 0.61, is the exponent.
• Dotted: if every token were a new word. The curve stays below it, but never lies down.
• 11 thousand tokens → 3,279 words; 100 thousand → 18,692; 1 million → 63,306.

Zipf's law: a few words are everywhere

Predict. Of the 101,631 distinct words, what share occurs exactly once in the whole corpus: 5%, 25%, or more than half?

how to read it

• Each point is a word. x axis: its rank (1 = most frequent); y axis: how many times it occurs. Both on a log scale.
• At the top, "the" (106,381 times), "of", "is": the first 10 are 21% of all tokens.
• In the middle, "car" (rank 271, 799 times) and "hockey" (rank 625, 374 times).
• Red band: 55% of the words occur exactly once.

Simulator: tokenize a paragraph

Predict. Switch from unigrams to uni + bi, what ngram_range=(1, 2) does: does the vocabulary grow a little, double, or more?

A map of the approaches

Once the text is cut into tokens, four pages turn the tokens into vectors, from the simplest to the most powerful. Point at a family to follow it.

What each one does, and cannot do

Bag-of-words: a document is its counts

       cachorro  comeu  do  dorme  gato  peixe  ração  sonha
doc 1         0      1   0      0     1      1      0      0
doc 2         1      1   1      0     1      0      1      0
doc 3         0      0   0      1     2      0      0      1
  

• One row per document, one column per word; "gato" is 2 in document 3. Word order is thrown away.
• 13 of the 24 cells are already zero. The 11,314 posts: \(11{,}314 \times 101{,}631\), 99.90% zeros, stored in a sparse format.

TF-IDF: counts weighted by rarity

then each row is divided by its length. Predict. In document 1 (gato, comeu, peixe: one each), which word weighs most?

     cachorro   comeu      do   dorme  gato   peixe   ração   sonha
idf    1.6931  1.2877  1.6931  1.6931   1.0  1.6931  1.6931  1.6931
       cachorro  comeu     do  dorme   gato  peixe  ração  sonha
doc 1     0.000  0.548  0.000  0.000  0.425   0.72  0.000  0.000
doc 2     0.505  0.384  0.505  0.000  0.298   0.00  0.505  0.000
doc 3     0.000  0.000  0.000  0.543  0.641   0.00  0.000  0.543
  

"peixe", only in document 1, weighs 0.720; "gato", in all three, 0.425. Same zeros, each row of length 1. Spärck Jones (1972); Salton & Buckley (1988).

Similarity and distance: four measures

A query and two candidates

Columns: cachorro, comeu, do, dorme, gato, peixe, ração, sonha. The query is A.

Why the cosine is the default for text

• Cosine: A3 is identical to A (1.000), B a partial match (0.471); B scores 0.471 against A and A3 alike.
• Euclidean gets it backwards: A3 is farther from A (3.464) than B (2.236), only because it is longer.
• Dot product: B against A3 scores 6, three times its 2 against A, for the same content.
• Jaccard agrees here, but it is blind to how often a word appears.

The limits of sparse representations

• Synonyms score zero: cosine 0.00. Every word is its own axis, orthogonal to the others.
• Word order is invisible: "the dog bit the man" and "the man bit the dog" have cosine 1.00.
• One spelling, two meanings: manga (sleeve or mango) makes unrelated sentences look alike, and two sentences with the same meaning look unrelated (0.14 in both cases).
• New words vanish: a word not seen in fit has no column.

Dense: know a word by its company

"You shall know a word by the company it keeps" (Firth, 1957)

• Distributional hypothesis (Harris, 1954): words used in similar contexts have similar meanings.
• "car" and "automobile" rarely share a sentence, but both appear next to engine, dealer, tires.
• Each word or text gets a few hundred numbers, learned so that similar contexts give nearby vectors.

Word embeddings, sentence embeddings

Word embeddings: what a vector looks like

GloVe, trained on 6 billion tokens of English text: 400,000 words, 50 numbers each.

glove = api.load('glove-wiki-gigaword-50')
print('cat =', glove['cat'][:8].round(3), '...')
print(glove.most_similar('cat', topn=5))
  

cat = [ 0.453 -0.501 -0.537 -0.016  0.222  0.546 -0.673 -0.689] ... (50 numbers, none of them zero)
nearest to cat: dog 0.92, rabbit 0.85, monkey 0.80, rat 0.79, cats 0.79
cosine car / automobile: 0.70
cosine car / banana: 0.22
  

No single number means anything; the neighbors do. "car" and "automobile" never shared a bag-of-words column; here they have cosine 0.70.

Sentence embeddings: one vector per sentence

paraphrase-multilingual-MiniLM-L12-v2, with normalize_embeddings=True: 384 numbers per text, 50+ languages.

(3, 384)
o carro é rápido = [ 0.016  0.033 -0.08   0.037 -0.014 -0.002  0.039 -0.028] ... (384 numbers)
vs 1: cosine 0.97 · dot 0.97 · Euclidean 0.25
vs 2: cosine 0.23 · dot 0.23 · Euclidean 1.24
  

At length 1, dot product = cosine, and the Euclidean distance tells the same story in reverse.

Sparse vs dense: one sentence, three vectors

"the cat sleeps on the couch": bag-of-words over the newsgroups' vocabulary, the average of its GloVe vectors, its sentence embedding.

What "sparse" and "dense" mean

Simulator: four ways to turn a sentence into a vector

Predict. On the paraphrase "the car is fast" / "this automobile goes quickly", which of the four gives cosine 0? Then swap the word order and tick B three times.

Sparse vs dense: the same pairs, both ways

TF-IDF fitted on the sentences of the table; a small multilingual sentence-transformers model (384 numbers per text).

Reading the table, row by row

Where TF-IDF is stronger: exact rare tokens

Two help-desk entries that differ only in an error code (0x80070005 vs 0x80004005), and a query for each code.

Classification: how many labels do you have?

Four newsgroups, a logistic regression used as a black box; only the representation changes. Accuracy on 1,364 later posts (guessing one class: about 25%).

Why: what each one arrives knowing

Sparse vs dense, side by side

Summary: which one, when

1. Start with TF-IDF: cheap and hard to beat when labels and domain vocabulary are plentiful.
2. Reach for embeddings when meaning matters more than exact words (search, short texts, paraphrases, several languages) or labels are scarce.
3. When exact tokens and meaning matter, combine them.

Sparse vs dense on real posts

• Four newsgroups (computer graphics, baseball, electronics, Christianity), about 2,100 posts, represented twice: TF-IDF and sentence embeddings.
• Steps: load the posts · two representations · which one finds posts about the same subject? · semantic vs keyword search · look at the two spaces · the same black-box classifier on both.
• Download the lab notebook and open it in Jupyter or Colab; in Colab, run !pip install sentence-transformers first.

Each of the four sub-pages (U4.1 to U4.4) has its own simulator and lab.