Machine Learning · Insper · E3.3

BERTopic

Fundamental approach

What if a topic is a cluster of documents?

The path of this part

Why embeddingsshort documents share few words
Embed and reduce384 numbers per post, then UMAP to 5
ClusterHDBSCAN; outliers are topic −1
Describec-TF-IDF, the cluster as a document
In one callthe BERTopic class, same partition

Each stage answers one question: why are word counts not enough? → where does meaning become distance? → where are the dense regions? → what words name a region? → does the library do the same?

Stage 1 · why are word counts not enough?

Short documents: where word counts run out

LDA (latent Dirichlet allocation) and NMF (non-negative matrix factorization) see only word counts. With the first 20 words of each post, ARI (adjusted Rand index) with the forums: NMF 0.12, LDA 0.03.

Predict. "I want my money back" / "please process a refund": cosine of their TF-IDF vectors?

Meaning as distance

TF-IDF cosine

0.00

no shared word

embedding cosine

0.60

a sentence embedding does not need shared words

On the 20-word posts, plain k-means on the embeddings reaches ARI 0.582, five times NMF's.

The idea behind BERTopic: first place the documents where meaning is distance, then look for groups.

Stage 2 · where does meaning become distance?

BERTopic: four steps, each a lesson you know

Step 1 — embed

Each post → 384 numbers with all-MiniLM-L6-v2 (a small English sentence-transformer, fast on a CPU).

Predict. For each post, take its 10 nearest posts. What share comes from the same forum, with TF-IDF and with embeddings?

TF-IDF

83.2%

embeddings

94.2%

Step 2 — reduce with UMAP

In 384 dimensions distances concentrate: density-based clustering suffers.
UMAP (uniform manifold approximation and projection) to 5 dimensions, not 2: the goal is to cluster, not to draw.
min_dist=0.0 packs similar points tightly, which a density clusterer wants.

Same-forum neighbors after UMAP: 95.4%. The neighborhoods survived. UMAP is stochastic: fix random_state.

Stage 3 · where are the dense regions?

Step 3 — cluster with HDBSCAN

HDBSCAN (hierarchical density-based clustering) finds dense regions of any shape; one main knob, min_cluster_size=10.

Predict. Nobody sets the number of clusters. How many will there be, and will every post get one?

clusters

from min_cluster_size and the density

outliers (−1)

431

20% of the posts

pure

21 / 24

≥ 90% of one forum

Hockey (516) and religion (554) are one big cluster each; graphics and space break into many small ones (file formats, graphics cards, the Moon…).

Simulator: `min_cluster_size`, one value at a time

Predict. Raise min_cluster_size from 2 to 40: will the number of topics fall smoothly?

Stage 4 · what words name a region?

Step 4 — describe each cluster with c-TF-IDF

Join the documents of a cluster into one pseudo-document, then a class-based TF-IDF (c-TF-IDF):

\[ \text{c-TF-IDF}(t, c) = \frac{n_{t,c}}{\sum_{t'} n_{t',c}} \times \log\!\Big(1 + \frac{A}{f_t}\Big) \]

\(n_{t,c}\): count of word \(t\) in cluster \(c\); the denominator: all the words of \(c\)
\(f_t\): count of \(t\) over all clusters · \(A\): average words per cluster

TF-IDF with the cluster as the document: a word scores high when it is frequent in this cluster and rare in the others.

c-TF-IDF by hand, step by step

The nine single-theme sentences as three clusters. Predict. team appears 3 times, game twice, both only in sports: which gets the larger weight, and which the larger score?

Stage 5 · does the library do the same?

All four steps in one call

The same objects and the embeddings of step 1 in BERTopic(...): same partition (ARI 1.0), renumbered by size.

topic	posts	Name
−1	431	`-1_space_jpeg_launch_image`
0	567	`0_god_jesus_people_church`
1	520	`1_team_game_hockey_play`
2	66	`2_image_graphics_package_processing`
3	55	`3_card_mode_vesa_vga`
4	52	`4_moon_lunar_prize_money`
5	43	`5_space_nasa_shuttle_astronaut`
6	41	`6_van_het_orbit_een`

The English stop-word list of the vectorizer decides which words can name a topic.

Summary

Short documents share few words; a sentence embedding puts same-meaning documents close anyway.
Four independent steps: embed → UMAP → HDBSCAN → c-TF-IDF.
HDBSCAN sets the number of topics; sparse posts go to topic −1.
c-TF-IDF names a cluster by words frequent in it and rare elsewhere.

Practice

To close the fundamental part

Why does BERTopic reduce the embeddings to 5 dimensions and not 2?
What does topic −1 hold, and why do LDA and NMF have nothing like it?
Why is game weighted more than team, and still scores less in sports?

In the lab: steps 4 to 7 and the simulation the embeddings or HDBSCAN? (overview page). Next: the fundamental slides of E3.4 Comparing the Models.