Machine Learning · Insper · E2.1
Bag-of-Words
Fundamental approach
How a document becomes a row of counts, and how alike two rows are
The idea
a column
for each word of the vocabulary
a cell
how many times that word occurs in that document
Nothing is learned except the vocabulary: no model, no training.
The path of this part
- Count the wordsthe count matrix, CountVectorizer, counts or presence
- Compare two bagscosine similarity: the angle, not the distance
- What the bag losesword order; bigrams buy part of it back
Each stage answers one question: how does a document become a row of numbers? → how alike are two documents? → what does the bag throw away?
Stage 1 · how does a document become a row of numbers?
Three documents, cut into tokens
"O gato comeu o peixe." ['gato', 'comeu', 'peixe']
"O cachorro comeu a ração do gato." ['cachorro', 'comeu', 'ração', 'do', 'gato']
"Gato dorme, gato sonha." ['gato', 'dorme', 'gato', 'sonha']
the cat ate the fish · the dog ate the cat's food · cat sleeps, cat dreams
Predict. How many columns will the matrix have? And what number goes in the cell (d3, gato)?
CountVectorizer builds the matrix
bow = CountVectorizer()
X = bow.fit_transform(docs)
print(pd.DataFrame(X.toarray(), columns=bow.get_feature_names_out(), index=['d1', 'd2', 'd3']))
print(type(X).__name__, X.shape, 'non-zeros:', X.nnz)
cachorro comeu do dorme gato peixe ração sonha
d1 0 1 0 0 1 1 0 0
d2 1 1 1 0 1 0 1 0
d3 0 0 0 1 2 0 0 1
csr_matrix (3, 8) non-zeros: 11
\[ \text{doc}_i \;\longrightarrow\; x_i \in \mathbb{R}^{|V|}, \qquad x_{ij} = \text{times word } j \text{ occurs in document } i \]
8 columns, the vocabulary \(V\) in alphabetical order; (d3, gato) is 2. "Bag" is literal: pour the words into a bag, shake it, and only the counts survive.
fit learns the vocabulary, transform only applies it
The vectorizer above was fitted on the three documents. Now three new ones:
- "O gato comeu o gato." (the cat ate the cat)
- "O cachorro latiu." (the dog barked)
- "Um felino adormeceu." (a feline fell asleep)
Predict. What happens to "latiu", and what vector does "Um felino adormeceu." become?
Unseen words have no column
[[0 1 0 0 2 0 0 0]
[1 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0]]
- "latiu" is silently dropped.
- "um felino adormeceu" is made only of unseen words: the zero vector.
A vectorizer is a fitted step, like a scaler: fit it on the training documents only. Words that exist only in the test set must not shape the columns.
Counts or presence: binary=True
[[0 1 0 0 1 1 0 0]
[1 1 1 0 1 0 1 0]
[0 0 0 1 1 0 0 1]]
d3's 2 under gato became 1.
short texts
titles, tweets, queries: a word rarely repeats, so both versions are almost identical
long documents
counts let one word that a post repeats many times dominate its vector
Stage 2 · how alike are two documents?
Comparing two bags: the cosine
\[ \cos(x_a, x_b) = \frac{x_a \cdot x_b}{\lVert x_a \rVert\, \lVert x_b \rVert} \]
For counts: 0 (no word in common) to 1 (the same words in the same proportions).
Predict. d1 shares comeu and gato with d2, and only gato with d3 (which has it twice). Which pair is more similar?
Counting and comparing, step by step
The three documents: each of the 12 tokens adds one to its cell, then each pair is compared by the cosine. Predict. Which pair is the most similar, and by how much?
Why the angle and not the distance?
Paste d1 twice and compare the result with d1:
"O gato comeu o peixe." [0 1 0 0 1 1 0 0]
"O gato comeu o peixe. O gato comeu o peixe." [0 2 0 0 2 2 0 0]
Predict. What is their Euclidean distance? And their cosine?
Euclidean distance
1.732
\(\sqrt{3}\): grows with the difference in length
cosine
1.000
looks only at the direction
The length of a document is not what it is about
That is why nearest neighbors and clustering on text use metric='cosine', or first divide each row by its length (which the TF-IDF vectorizer, for term frequency × inverse document frequency, does by default).
Stage 3 · what does the bag throw away?
Word order is lost
- "O cachorro mordeu o homem." (the dog bit the man)
- "O homem mordeu o cachorro." (the man bit the dog)
Predict. What is the cosine between these two sentences?
| features | vectors | cosine |
|---|
| single words (3 columns) | [1, 1, 1] and [1, 1, 1] | 1.00 |
| + bigrams (7 columns) | "cachorro mordeu", "mordeu homem" only in the first; "homem mordeu", "mordeu cachorro" only in the second | 0.60 |
Bigrams buy back local order, at a price
- With single words the two sentences are the same vector: who bit whom is gone.
ngram_range=(1, 2) adds pairs of neighboring tokens and the cosine falls to 0.60.- The price: more columns, most of them rare.
The bigram joins tokens that are neighbors after tokenization: "o" was dropped, so "mordeu o homem" gave "mordeu homem".
Simulator: build the bag yourself
Predict. Load the word-order preset: will the first two sentences get different vectors? Then tick add bigrams.
Summary
- A document becomes its word counts, one column per vocabulary word; word order is thrown away.
fit fixes the vocabulary; unseen words are dropped. Fit on the training documents only.binary=True keeps only presence, which matters for long documents.- Compare documents with the cosine: length is not subject.
- Order is lost: "the dog bit the man" = "the man bit the dog". Bigrams buy back local order, with many more columns.
Fundamental lab · about 25 min, in pairs
Bag-of-words on real posts, steps 1 to 5
- Four newsgroups (computer graphics, baseball, electronics, Christianity), about 2,100 posts; scikit-learn only.
- One question: are the most similar posts about the same subject? The neighbor agreement answers it (0.251 by chance).
- Steps: load the posts · the count matrix · what each newsgroup talks about · shrinking the vocabulary · counts, presence, stop words.
- At every step, answer the Predict before running. Simulation: a stop list learned from the data (step 5).
- Download the lab notebook and open it in Jupyter or Colab.
Pace: steps 1 and 2 in 5 min; steps 3 and 4 in 8 min; step 5 and the simulation in 12 min.