The path of this part
- Count the vocabularyHeaps' and Zipf's laws
- Merge the variantsstems and lemmas, with NLTK and spaCy
- Never an unknown wordWordPiece, SentencePiece; who decides what
Each stage answers one question: how big does the vocabulary get? → can the variants of a word share one column? → how can no word be unknown?
Stage 1 · how big does the vocabulary get?
How big does the vocabulary get?
The 20 Newsgroups collection: 11,314 posts from Usenet groups of the early 1990s. Read them one token at a time and count the distinct words seen so far.
Predict. After a few hundred thousand tokens, does the number of distinct words level off, or keep growing?
Heaps' law: the vocabulary never stops growing
how to read it
- x axis: tokens read so far; y axis: distinct words seen. Both on a log scale.
- On a log scale a power law is a straight line; its slope, 0.61, is the exponent.
- Dotted: if every token were a new word. The curve stays below it, but never lies down.
- 11 thousand tokens → 3,279 words; 100 thousand → 18,692; 1 million → 63,306.
Ten times more text gives about 4 times more words, with no end: a vocabulary built today meets new words tomorrow, and a sparse representation has no column for them.
Zipf's law: a few words are everywhere
Predict. Of the 101,631 distinct words, what share occurs exactly once in the whole corpus: 5%, 25%, or more than half?
how to read it
- Each point is a word. x axis: its rank (1 = most frequent); y axis: how many times it occurs. Both on a log scale.
- At the top, "the" (106,381 times), "of", "is": the first 10 are 21% of all tokens.
- In the middle, "car" (rank 271, 799 times) and "hockey" (rank 625, 374 times).
- Red band: 55% of the words occur exactly once.
Most columns of a bag-of-words describe a single document; the few words at the top are precisely the stop words.
Simulator: tokenize a paragraph
Predict. Switch from unigrams to uni + bi, what ngram_range=(1, 2) does: does the vocabulary grow a little, double, or more?
Stage 2 · can the variants of a word share one column?
The same sentence, normalized
Stemming chops suffixes by rules (the Portuguese RSLPStemmer of NLTK, the Natural Language Toolkit; RSLP: Removedor de Sufixos da Língua Portuguesa); lemmatization returns the dictionary form (spaCy). The sentence is the one of the fundamental part.
Predict. A stemmer cuts "comeu" (ate) to "com". Which other, unrelated words could land in the same column?
| step | output |
|---|---|
| stop words out | gatos · não · comeram · ração · cachorro · mas · gato · comeu |
| stems (RSLP) | os · gat · não · com · a · raç · do · cachorr · mas · o · gat · com |
| lemmas (spaCy) | o · gato · não · comer · o · ração · de o · cachorro · mas · o · gato · comeu |
| bigrams | os gatos · gatos não · não comeram · comeram ração · ração do · do cachorro · cachorro mas · mas gato · gato comeu |
- "com" (with) and "comida" (food) also stem to "com": four unrelated words share one column.
- The small lemmatizer left "comeu" as it is: it is a statistical model, and it makes mistakes.
- "não" survives this stop list; scikit-learn's English list removes "not", "no", "never".
Stage 3 · how can no word be unknown?
Subwords: the tokenizers of neural models
Frequent words stay whole, rare words split into frequent pieces: no word is ever unknown.
Predict. An English tokenizer meets "não" and "cachorro". What does it make of them?
| tokenizer | output |
|---|---|
WordPiece, all-MiniLM-L6-v230,522 English pieces | tokenization → token ##ization unbelievably → un ##bel ##ie ##va ##bly s → os ga ##tos na ##o come ##ram a ra ##cao do ca ##cho ##rro , mas o ga ##to come ##u ! |
| SentencePiece, multilingual model 250,002 pieces, 50+ languages | s → ▁Os ▁gato s ▁não ▁come ram ▁a ▁ra ção ▁do ▁cachorro , ▁mas ▁o ▁gato ▁come u ! desparafusaram (they unscrewed) → ▁des para fus aram |
##: glued to the piece before; ▁: start of a word. Each piece has its own vector, so a word never seen whole still gets one.
Who decides what: each tokenization decision, by family
| bag-of-words, TF-IDF term frequency × inverse document frequency | word embeddings (GloVe) | sentence embeddings | |
|---|---|---|---|
| lowercase, punctuation | your choice (scikit-learn's default lowercases and drops punctuation) | GloVe's vocabulary is all lowercase | the model's tokenizer decides |
| stop words | optional: removes columns | optional, before averaging | no: "não" changes the meaning |
| stemming, lemmatization | optional: merges columns | no: one vector per form | no |
| n-grams | optional: columns for pairs | no | no: the model reads the sentence in order |
| subwords | no | no: a word outside the vocabulary gets no vector | yes, the model's tokenizer |
Normalizing is a decision of the sparse family. In dense embeddings the tokenizer and the vocabulary come with the model: the text goes in as it is.
Summary
- Heaps' law: the vocabulary grows like \(N^{0.61}\) and never stops; Zipf's law: 55% of the words occur once.
- Stemming chops suffixes (and merges unrelated words); lemmatization looks up the dictionary form (and makes mistakes).
- Subword tokenizers split rare words into known pieces: no word is ever unknown.
- Normalizing is a decision of the sparse family; dense models bring their own tokenizer.
Advanced lab · about 20 min, in pairs
Two spaces and a classifier, steps 5 and 6
- Continue in the same notebook and session as steps 1 to 4.
- Step 5: t-SNE (t-distributed stochastic neighbor embedding) maps of the two spaces and their silhouette.
- Step 6: the same black-box classifier on both representations, with all the labels and with 10 per class.
- Simulation: does order matter? (step 5).
- The lab notebook, from step 5.
Pace: step 5 and the simulation in 12 min; step 6 in 8 min.