Text Representation

Everything we have done so far assumed data arrives as a numeric table. Text does not. This lesson builds the bridge — from raw strings to vectors — that every text application depends on, and paves the way for Topic Modeling & BERTopic.

The central question: how do we turn a document into a vector so that geometric tools (distances, clustering, classifiers) apply?

Tokenization and the vocabulary

The first step is splitting text into units — tokens (words, subwords, or characters) — and building a vocabulary: the set of distinct tokens across the corpus. Common normalizations: lowercase, strip punctuation, optionally remove stop words ("the", "of", "and" — frequent words carrying little content) and reduce words to stems or lemmas ("running" → "run").

Bag-of-Words

The classical representation (a direct descendant of 1950s information retrieval): represent a document by its word counts, ignoring order entirely.

from sklearn.feature_extraction.text import CountVectorizer
vec = CountVectorizer()
X = vec.fit_transform(corpus)   # sparse matrix: n_docs × |V|

"Bag" is literal: "the dog bit the man" and "the man bit the dog" get identical vectors. Word order — and thus most syntax — is discarded.

TF-IDF

Raw counts overweight common words. TF-IDF (term frequency × inverse document frequency, Spärck Jones 1972) rescales each count by how distinctive the word is across the corpus:

where \(n\) is the number of documents and \(\text{df}(t)\) is the number of documents containing \(t\). A word appearing in every document (idf ≈ 1 after smoothing) is discounted; a word concentrated in few documents is amplified. scikit-learn then normalizes each document vector to unit length.

Note how "the" — present in three of four documents — gets low weights, while distinctive words like "cheese" and "learning" score high in their documents. Documents are compared with cosine similarity:

n-grams: buying back a little order

Counting sequences of \(n\) consecutive tokens (bigrams: "not good", "machine learning") recovers local word order:

TfidfVectorizer(ngram_range=(1, 2))   # unigrams + bigrams

The cost: the vocabulary — and the dimensionality — explodes combinatorially.

The limits of sparse representations

Bag-of-words/TF-IDF vectors are sparse (mostly zeros), high-dimensional (\(|V|\) can be 10⁵+), and — critically — treat words as atomic symbols:

• "car" and "automobile" are orthogonal dimensions: similarity zero, despite being synonyms;
• "bank" (river) and "bank" (finance) are the same dimension: context is invisible;
• a document about dogs and one about puppies may share no vocabulary and be judged unrelated.

These are exactly the failures that motivated dense embeddings.

Dense embeddings

Word embeddings — word2vec intuition

word2vec (Mikolov et al., 2013) learns a dense vector (~300 dims) per word by training a shallow network to predict words from their contexts. The distributional hypothesis does the magic: words appearing in similar contexts get similar vectors. Synonyms land close together, and directions encode relations — the famous

GloVe (2014) and fastText (2016) refine the idea. Limitation: one vector per word, so "bank" still has a single meaning averaged over all its uses.

Sentence embeddings — contextual and whole-document

Transformer models (BERT, 2018) produce contextual embeddings — the vector for "bank" differs in "river bank" vs "bank loan". Sentence-BERT (Reimers & Gurevych, 2019) fine-tunes such models so that a whole sentence or short document maps to a single dense vector (~384–768 dims) where cosine similarity ≈ semantic similarity:

# pip install sentence-transformers
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')
emb = model.encode(["The dog chased the cat",
                    "A hound pursued a feline",
                    "Interest rates rose again"])
# emb.shape == (3, 384); emb[0]·emb[1] high, emb[0]·emb[2] low

The first two sentences share almost no vocabulary yet get highly similar vectors — precisely what TF-IDF cannot do.

Sparse vs dense: when to use which

Quiz