Machine Learning · Insper · E3
Topic Modeling & BERTopic
Fundamental approach
Given thousands of documents, what are they about?
A question that sounds simple
- Nobody labels the documents, nobody lists the themes: unsupervised, the text-world sibling of clustering.
- Uses: a year of reviews or support tickets, a news archive, social media, a scientific literature.
LDA
latent Dirichlet allocation: a probabilistic story of how words are written
NMF
non-negative matrix factorization of the TF-IDF matrix (term frequency × inverse document frequency)
BERTopic
sentence embeddings + UMAP (a nonlinear projection) + HDBSCAN (density-based clustering) + a variant of TF-IDF
Two corpora: 11 hand-written sentences, and 2,155 real posts from four forums (the forum is hidden from the models, to check them).
The path of this lesson
- What is a topic?words that go together, or documents that go together
- LDALatent Dirichlet Allocation: a story of how words get written
- NMFnon-negative matrix factorization: a document as a sum of parts
- BERTopicembed, reduce, cluster, describe
- Compare and choosethe three side by side
Each stage answers one question: what are we looking for in the documents? → what story could have written these words? → can a matrix factorization find the same topics? → what if a topic is a cluster of documents? → which model should I trust, and when?
Stage 1 · what are we looking for in the documents?
What is a topic? Two answers
a distribution over words
"Sports" gives high probability to team, game, goal, almost none to oven. A document is a mixture: 80% sports, 20% space.
LDA, and NMF with non-negative weights
a cluster of documents
Each document belongs to one topic, or to none; the topic's words are computed afterwards, to describe the cluster.
BERTopic
The hand-made corpus, as counts
| doc | team | game | goal | rocket | launch | moon | bread | bake |
|---|
| 0 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 |
| 3 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 |
| 6 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 |
| 9 | 1 | 1 | 0 | 1 | 1 | 0 | 0 | 0 |
| 10 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 0 |
11 sentences, 53 words kept, a vocabulary of 33 (English stop words removed).
Three themes: sports, space, cooking; the last two sentences mix them.
Predict. A topic model receives only this matrix, without the comments. Can it find the three themes?
Five pages, two approaches each
| page | a topic is | fundamental | advanced |
|---|
| E3.1 LDA | a distribution over words | the story by hand, the prior, the posts | known topics recovered; too few words |
| E3.2 NMF | a row of weights ≥ 0 | \(X \approx WH\), against the SVD (singular value decomposition), the posts | how many topics: perplexity, coherence |
| E3.3 BERTopic | a cluster of documents | short texts; the four steps; one call | names by hand; tuning the words |
| E3.4 Comparing | | the three side by side | the pitfalls |
The path goes through the fundamental slides of E3.1 to E3.4, in order, then comes back here for the summary and the lab.
Summary
- A topic is a distribution over words (LDA, NMF) or a cluster of documents (BERTopic).
- LDA: every word slot draws a topic, then a word: \(p(w \mid d) = \sum_k \theta_{d,k}\, \varphi_{k,w}\); \(\alpha\) sets how mixed a document is.
- NMF: \(X \approx WH\) with nothing negative; each topic a whole theme.
- BERTopic: embed → UMAP → HDBSCAN → c-TF-IDF (TF-IDF per cluster); the number of topics comes from the data, outliers are topic −1.
- No model wins everywhere: measure against something the models never saw, and read the topics.
| what you see | what to do |
|---|
| LDA and NMF near chance on short texts | embed the documents (BERTopic) |
| topics named the, of, to | a stop-word list in the vectorizer |
Fundamental lab · about 40 min, in pairs
Topics in five forums, three ways: steps 1 to 7
- Five other forums of 20 Newsgroups: 2,613 posts, between 495 and 551 per forum; the forum stays hidden from the models.
- Steps 2–3: LDA and NMF. Steps 4–7: BERTopic one step at a time, then in one call.
- Download the lab notebook and open it in Jupyter or Colab; in Colab run
!pip install bertopic first.
- At every step, answer the Predict before running.
- Simulations: more topics for NMF (step 3) and the embeddings or HDBSCAN? (step 6).
Pace: steps 1 to 3 and the first simulation in 15 min; steps 4 and 5 in 10 min; steps 6 and 7 and the second simulation in 15 min.