The idea
a column
for each word of the vocabulary
a row
for each document
a cell
how many times that word occurs in that document
Nothing is learned except the vocabulary: no model, no training.
The path of this lesson
- Count the wordsthe count matrix, CountVectorizer, counts or presence
- Compare two bagscosine similarity: the angle, not the distance
- What the bag losesword order; word and character n-grams
- A real corpussparsity, and what raw counts put first
Each stage answers one question: how does a document become a row of numbers? → how alike are two documents? → what does the bag throw away? → what happens on a real corpus?
Stage 1 · how does a document become a row of numbers?
Three documents, cut into tokens
"O gato comeu o peixe." ['gato', 'comeu', 'peixe']
"O cachorro comeu a ração do gato." ['cachorro', 'comeu', 'ração', 'do', 'gato']
"Gato dorme, gato sonha." ['gato', 'dorme', 'gato', 'sonha']
the cat ate the fish · the dog ate the cat's food · cat sleeps, cat dreams
Predict. How many columns will the matrix have? And what number goes in the cell (d3, gato)?
Counts, by hand
The vocabulary, sorted alphabetically: 8 words.
| cachorro | comeu | do | dorme | gato | peixe | ração | sonha | |
|---|---|---|---|---|---|---|---|---|
| d1 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 |
| d2 | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 0 |
| d3 | 0 | 0 | 0 | 1 | 2 | 0 | 0 | 1 |
CountVectorizer builds the same matrix
bow = CountVectorizer()
X = bow.fit_transform(docs)
print(type(X).__name__, X.shape, 'non-zeros:', X.nnz)
print(bow.vocabulary_)
csr_matrix (3, 8) non-zeros: 11
{'gato': 4, 'comeu': 1, 'peixe': 5, 'cachorro': 0, 'ração': 6, 'do': 2, 'dorme': 3, 'sonha': 7}
"Bag" is literal: pour the words into a bag, shake it, and only the counts survive. Salton, Wong & Yang (1975): documents are points in a space with one axis per word.
fit learns the vocabulary, transform only applies it
The vectorizer above was fitted on the three documents. Now three new ones:
- "O gato comeu o gato." (the cat ate the cat)
- "O cachorro latiu." (the dog barked)
- "Um felino adormeceu." (a feline fell asleep)
Predict. What happens to "latiu", and what vector does "Um felino adormeceu." become?
Unseen words have no column
[[0 1 0 0 2 0 0 0]
[1 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0]]
- "latiu" is silently dropped.
- "um felino adormeceu" is made only of unseen words: the zero vector.
A vectorizer is a fitted step, like a scaler: fit it on the training documents only. Words that exist only in the test set must not shape the columns.
Counts or presence: binary=True
[[0 1 0 0 1 1 0 0]
[1 1 1 0 1 0 1 0]
[0 0 0 1 1 0 0 1]]
d3's 2 under gato became 1.
short texts
titles, tweets, queries: a word rarely repeats, so both versions are almost identical
long documents
counts let one word that a post repeats many times dominate its vector
Stage 2 · how alike are two documents?
Comparing two bags: the cosine
\[ \cos(x_a, x_b) = \frac{x_a \cdot x_b}{\lVert x_a \rVert\, \lVert x_b \rVert} \]For counts: 0 (no word in common) to 1 (the same words in the same proportions).
Predict. d1 shares comeu and gato with d2, and only gato with d3 (which has it twice). Which pair is more similar?
By hand, then checked
d1, d2: \(x_1 \cdot x_2 = 1 + 1 = 2\)
\[ \cos = \frac{2}{\sqrt{3}\,\sqrt{5}} = 0.516 \]d1, d3: only gato, count 2 in d3
\[ \cos = \frac{2}{\sqrt{3}\,\sqrt{6}} = 0.471 \]
d1 d2 d3
d1 1.000 0.516 0.471
d2 0.516 1.000 0.365
d3 0.471 0.365 1.000
cosine_similarity(X)
Why the angle and not the distance?
Paste d1 twice and compare the result with d1:
"O gato comeu o peixe." [0 1 0 0 1 1 0 0]
"O gato comeu o peixe. O gato comeu o peixe." [0 2 0 0 2 2 0 0]
Predict. What is their Euclidean distance? And their cosine?
Euclidean distance
1.732
\(\sqrt{3}\): grows with the difference in length
cosine
1.000
looks only at the direction
The length of a document is not what it is about
That is why nearest neighbors and clustering on text use
metric='cosine', or first divide each row by its length (which the TF-IDF vectorizer does by default).Stage 3 · what does the bag throw away?
Word order is lost
- "O cachorro mordeu o homem." (the dog bit the man)
- "O homem mordeu o cachorro." (the man bit the dog)
Predict. What is the cosine between these two sentences?
| features | vectors | cosine |
|---|---|---|
| single words (3 columns) | [1, 1, 1] and [1, 1, 1] | 1.00 |
| + bigrams (7 columns) | "cachorro mordeu", "mordeu homem" only in the first; "homem mordeu", "mordeu cachorro" only in the second | 0.60 |
Bigrams buy back local order, at a price
- With single words the two sentences are the same vector: who bit whom is gone.
ngram_range=(1, 2)adds pairs of neighboring tokens and the cosine falls to 0.60.- The price: more columns, most of them rare.
The bigram joins tokens that are neighbors after tokenization: "o" was dropped, so "mordeu o homem" gave "mordeu homem".
Character n-grams
analyzer='char_wb' cuts each word, padded with spaces, into overlapping pieces of 3 letters.
| pair | cosine | shared trigrams |
|---|---|---|
| sábado / sábados | 0.77 | ' sá', 'ado', 'bad', 'sáb', 'ába' |
| gato / gatos | 0.67 | ' ga', 'ato', 'gat' |
| gato / pato (cat / duck) | 0.50 | 'ato', 'to ' |
They tolerate plurals, conjugations and typos, with no stemmer. The cost: "gato" and "pato" share spelling, not meaning.
Stage 4 · what happens on a real corpus?
Sparsity: a real corpus
20 Newsgroups, training part: 11,314 posts, headers, signatures and quoted replies removed.
Predict. The vocabulary has 101,631 words. What fraction of the cells is zero?
zeros
99.90%
a median post uses 61 columns
as a dense array
9.2 GB
float64
as sparse CSR
13.3 MB
about 700 times less
Why vectorizers return sparse matrices
CSR (compressed sparse row) keeps only the non-zero values and their columns. Bigrams multiply the columns by 9, trigrams by 25; min_df=2 or max_features keep the size under control. .toarray() is for toy examples only.
What raw counts put first
Sum each column of the 11,314 posts and sort.
Predict. Which word comes first? And after removing the English stop words?
| vectorizer | top words | the leader |
|---|---|---|
| default | the, ax, to, of, and, in, is, that | "the": 106,381 times, in 9,475 posts |
stop_words='english' | ax, max, people, like, don, just, know, use | "ax": 62,387 times, in 20 posts |
Two problems, one cure
- Words in almost every post say nothing about the subject.
- "ax" repeats thousands of times in 20 posts with binary files pasted as text: one strange document dominates a column.
The cure, on the next page (U4.2 TF-IDF): weigh each word by how many documents contain it, not only by how often it occurs.
Simulator: build the bag yourself
Predict. Load the word-order preset: will the first two sentences get different vectors? Then tick add bigrams.
Summary
- A document becomes its word counts, one column per vocabulary word.
fitfixes the vocabulary; unseen words are dropped. Fit on training data only.- Compare documents with the cosine: length is not subject.
- Order is lost; bigrams and character n-grams buy some back, with many more columns.
- The matrix is 99.90% zeros: sparse storage is mandatory.
- Raw counts are led by words that are everywhere or in one strange document.
Lab · well under a minute, scikit-learn only
Bag-of-words on real posts
- Four newsgroups (computer graphics, baseball, electronics, Christianity), about 2,100 posts.
- One question: are the most similar posts about the same subject?
- Steps: load the posts · the count matrix · what each newsgroup talks about · shrinking the vocabulary · counts, presence, stop words · n-grams of words and characters · the bag cannot tell a post from its shuffled words.
- Download the lab notebook and open it in Jupyter or Colab.