Topic Modeling & BERTopic

A question that sounds simple

• Nobody labels the documents, nobody lists the themes: unsupervised, the text-world sibling of clustering.
• Uses: a year of reviews or support tickets, a news archive, social media, a scientific literature.

Two corpora: 11 hand-written sentences, and 2,155 real posts from four forums (the forum is hidden from the models, to check them).

The path of this lesson

What is a topic? Two answers

The hand-made corpus, as counts

LDA: a story of how the words got there

Latent Dirichlet Allocation (Blei, Ng & Jordan, 2003):

1. each topic \(k\) is a distribution \(\varphi_k\) over the vocabulary;
2. each document \(d\) has its own mixture \(\theta_d\), e.g. \((0.8, 0.2)\);
3. for every word slot: draw a topic \(z\) from \(\theta_d\), then a word from \(\varphi_z\).

Not a claim that people write this way. Word order plays no part: bag-of-words counts are enough.

The story by hand

A document 80% sports, 20% space:

\(p(\textit{team}) = 0.8 \times 0.40 + 0.2 \times 0.05 = 0.33\)

\(p(\textit{rocket}) = 0.8 \times 0.05 + 0.2 \times 0.40 = 0.12\)

Following the story, 4 of 12 slots drew the space topic, and they are exactly where moon, orbit, rocket, orbit appear.

The Dirichlet prior: how mixed is a document?

\(\alpha\)	largest share
0.1	0.89
1	0.62
10	0.42

In scikit-learn: doc_topic_prior and topic_word_prior, both \(1/k\) by default: few topics per document, few words per topic.

Simulator: write a document the LDA way

Predict. With \(\alpha = 0.1\), will the documents you draw come out in one color or in all three?

Fitting LDA: running the story backwards

• Generate a corpus with the story itself (300 documents, 30 words each), so the true topics are known.
• LDA recovers the true weights (0.35, 0.30, 0.25 on each topic's own words) within 0.01.
• The estimated mixtures correlate 0.97 with the true ones.

And on the 53 hand-written words?

• Seed 0: bread fresh game oven · team goal game orbit · rocket launch bread delayed: the themes mixed.
• Each document still goes confidently (0.86 to 0.88) to one topic.
• Over 20 seeds, only 1 separates the nine single-theme sentences cleanly.

LDA on 2,155 real posts

Four forums of 20 Newsgroups (graphics, hockey, space, religion); 5,725 words of 3+ letters in at least 5 posts and at most half of them.

NMF: \(X \approx WH\), with nothing negative

Non-negative matrix factorization (Lee & Seung, 1999): the same shape as PCA, plus the constraint that every entry is \(\ge 0\). Here \(X\) is TF-IDF.

NMF on the 11 sentences

Why non-negative? A comparison with SVD

Simulator: NMF parts versus SVD directions

Predict. From random weights, press step: will the NMF rows end up as whole themes or as contrasts like the SVD's? Will its error reach the SVD's?

NMF on the posts

One run on one corpus: the lab finds a corpus where the order is reversed.

How many topics, and are they any good?

No labels to cross-validate against: the score must come from the documents.

By hand: of 2,155 posts, 304 contain god, 159 jesus, 101 both; independent would be 1.04%, observed 4.69%: NPMI = 0.492.

Choosing \(k\)

At \(k = 4\): LDA coherence 0.224, diversity 0.85; NMF 0.369 and 1.00.

Perplexity only rises: read literally, it recommends \(k = 2\); and it does not track what people find interpretable.

Short documents: where word counts run out

First 20 words of each post: NMF 0.12, LDA 0.03.

Predict. "I want my money back" / "please process a refund": TF-IDF cosine?

Meaning as distance

On the 20-word posts, plain k-means on the embeddings reaches ARI 0.582, five times NMF's.

BERTopic: four steps, each a lesson you know

Stage 1 — embed

Each post → 384 numbers with all-MiniLM-L6-v2 (small, English, fast on a CPU).

Stage 2 — reduce with UMAP

• In 384 dimensions distances concentrate: density-based clustering suffers.
• UMAP to 5 dimensions, not 2: the goal is to cluster, not to draw.
• min_dist=0.0 packs similar points tightly, which a density clusterer wants.

Stage 3 — cluster with HDBSCAN

Hockey (516) and religion (554) are one big cluster each; graphics and space break into many small ones (file formats, graphics cards, the Moon…).

Simulator: min_cluster_size, one value at a time

Predict. Raise min_cluster_size from 2 to 40: will the number of topics fall smoothly?

Stage 4 — describe each cluster with c-TF-IDF

Join the documents of a cluster into one pseudo-document; then a class-based TF-IDF:

By hand, three clusters of the hand-made sentences: 16, 13 and 14 words, \(A = 14\).

BERTopic's own transformer gives exactly the same matrix.

The clusters of the posts, named

The class BERTopic chains the same four steps: same partition (ARI 1.0), same top words, only renumbered by size.

Stage 5 — tune the words, not the clusters

LDA vs NMF vs BERTopic, measured

Same 2,155 posts, 24 topics for all three, two seeds each. Predict first: whose topics are purest?

Reading the table

• Homogeneity: BERTopic 0.927, but on the 80% of posts it kept; with seed 43 it finds 16 broader topics and falls to 0.662.
• Coherence and diversity favor NMF: NPMI rewards co-occurring words, which NMF reconstructs; BERTopic picks distinctive words.
• Stability between seeds (ARI): LDA 0.164, NMF 0.736, BERTopic 0.324, with even the number of topics changing.

The overall comparison

Pitfalls: stop words and outliers

Pitfall: granularity is a cliff, not a dial

Scan a few values and read the topics; to merge topics without re-clustering, reduce_topics.

Pitfall: reproducibility

Add to the embeddings a noise of about one millionth (the size of CPU-vs-GPU differences), keep every seed:

Summary

1. A topic is a distribution over words (LDA, NMF) or a cluster of documents (BERTopic).
2. LDA: a generative story; needs many documents with enough words.
3. NMF: additive parts of the TF-IDF matrix; stable, cheap, coherent.
4. Choose \(k\) with coherence and by reading the topics; perplexity misleads.
5. BERTopic: embed → UMAP → HDBSCAN → c-TF-IDF; strong on short texts, explicit outliers, fragile to seeds and to min_cluster_size.

Topics in five forums, three ways

• Five other forums of 20 Newsgroups: 2,613 posts, between 495 and 551 per forum.
• Steps 2–3: LDA and NMF. Steps 4–7: BERTopic step by step, then in one call. Steps 8–10: inspect the topics, compare them with the forums, deal with the outliers.
• Download the lab notebook and open it in Jupyter or Colab; in Colab run !pip install bertopic first.
• Run the blocks in order, in one session; most of the time is step 4.