Machine Learning · Insper · E2.1

Bag-of-Words

Advanced approach

Pieces of words, a real corpus, and what raw counts put first

The path of this part

  1. Pieces of wordscharacter n-grams: plurals, typos, rhymes
  2. A real corpus99.90% zeros, CSR storage, the price of n-grams
  3. What raw counts put firstwords that are everywhere, one strange document
Each stage answers one question: what if we count pieces of words? → what does a real corpus look like as counts? → which words do raw counts put first?
Stage 1 · what if we count pieces of words?

Character n-grams

analyzer='char_wb' cuts each word, padded with spaces, into overlapping pieces of 3 letters.

paircosineshared trigrams
sábado / sábados0.77' sá', 'ado', 'bad', 'sáb', 'ába'
gato / gatos0.67' ga', 'ato', 'gat'
gato / pato (cat / duck)0.50'ato', 'to '
They tolerate plurals, conjugations and typos, with no stemmer. The cost: "gato" and "pato" share spelling, not meaning.
Stage 2 · what does a real corpus look like as counts?

Sparsity: a real corpus

20 Newsgroups, training part: 11,314 posts, headers, signatures and quoted replies removed.

Predict. The vocabulary has 101,631 words. What fraction of the cells is zero?

zeros

99.90%
a median post uses 61 columns

as a dense array

9.2 GB
float64

as sparse CSR

13.3 MB
about 700 times less

Why vectorizers return sparse matrices

Left: dots marking the non-zero cells of 200 posts by the 2,000 most frequent words, still 96.8% empty. Right: adding bigrams takes the vocabulary from 101,631 to 948,675 columns and trigrams to 2,524,547, while the median post fills only 61, 136 and 213 of them

CSR (compressed sparse row) keeps only the non-zero values and their columns. Bigrams multiply the columns by 9, trigrams by 25; min_df=2 or max_features keep the size under control. .toarray() is for toy examples only.

Stage 3 · which words do raw counts put first?

What raw counts put first

Sum each column of the 11,314 posts and sort.

Predict. Which word comes first? And after removing the English stop words?
vectorizertop wordsthe leader
defaultthe, ax, to, of, and, in, is, that"the": 106,381 times, in 9,475 posts
stop_words='english'ax, max, people, like, don, just, know, use"ax": 62,387 times, in 20 posts

Two problems, one cure

Left: the 15 most frequent words with the default tokenizer are function words led by the, 24% of all tokens. Right: with the English stop list, ax leads with 62,387 occurrences in only 20 posts
  • Words in almost every post say nothing about the subject.
  • "ax" repeats thousands of times in 20 posts with binary files pasted as text: one strange document dominates a column.
The cure, on the next page (E2.2 TF-IDF): weigh each word by how many documents contain it, not only by how often it occurs.

Summary

  1. Character n-grams match pieces of spelling: plurals and typos come close, and so do rhymes.
  2. A real count matrix is 99.90% zeros; CSR keeps only the non-zeros (13.3 MB instead of 9.2 GB).
  3. n-grams multiply the columns, and most of the new ones occur in a single post: min_df keeps them in check.
  4. Raw counts are led by words that are everywhere ("the") or by one strange document ("ax").
Next: TF-IDF (term frequency × inverse document frequency) weighs each word by how many documents contain it.
Advanced lab · about 15 min, in pairs

What the bag cannot see, steps 6 and 7

  • Continue in the same notebook and session as steps 1 to 5.
  • Step 6: n-grams of words and of characters, scored by the neighbor agreement.
  • Step 7: a post with its words shuffled has the very same count vector.
  • Simulation: how rare are n-grams? (step 6).
  • The lab notebook, from step 6.

Pace: step 6 and the simulation in 10 min; step 7 in 5 min.