Machine Learning · Insper · E3.4
Comparing the Models
Fundamental approach
Which model should I trust, and when?
The path of this part
- Measure the threesame posts, 24 topics, two seeds
- Compare overallwhat a topic is, short texts, outliers, cost
- Choosewhich model, for which corpus
Each stage answers one question: how do the three score on equal terms? → how do they differ beyond the scores? → which one for my corpus?
Stage 1 · how do the three score on equal terms?
LDA vs NMF vs BERTopic, measured
Same 2,155 posts, 24 topics for LDA (latent Dirichlet allocation), NMF (non-negative matrix factorization) and BERTopic, two seeds each. Homogeneity: 1 when every topic holds a single forum. Coherence: mean NPMI (normalized pointwise mutual information) of top words. Diversity: share of distinct top words. Predict: whose topics are purest?
| model | seed | topics | outliers | coherence | diversity | homogeneity |
|---|
| LDA | 42 | 24 | 0% | 0.265 | 0.72 | 0.526 |
| NMF | 42 | 24 | 0% | 0.391 | 0.92 | 0.646 |
| BERTopic | 42 | 24 | 20% | 0.251 | 0.88 | 0.927 |
| LDA | 43 | 24 | 0% | 0.276 | 0.70 | 0.574 |
| NMF | 43 | 24 | 0% | 0.407 | 0.91 | 0.644 |
| BERTopic | 43 | 16 | 10% | 0.295 | 0.82 | 0.662 |
Reading the table
- Homogeneity: BERTopic 0.927, but on the 80% of posts it kept; with seed 43 it finds 16 broader topics and falls to 0.662.
- Coherence and diversity favor NMF: NPMI rewards co-occurring words, which NMF reconstructs; BERTopic picks distinctive words.
- Stability between seeds, ARI (adjusted Rand index): LDA 0.164, NMF 0.736, BERTopic 0.324, with even the number of topics changing.
A coherence score measures one aspect of quality, not the truth.
Stage 2 · how do they differ beyond the scores?
The overall comparison
| LDA | NMF | BERTopic |
|---|
| a topic is | a distribution over words | a non-negative row of \(H\) | a cluster, named by c-TF-IDF |
| number of topics | fixed in advance | fixed in advance | found by HDBSCAN |
| short documents | weak | weak | strong |
| outliers | none | none | explicit (−1) |
| seed-dependence | strong | weaker (SVD start) | yes, even the number of topics |
| cost | cheap, CPU | cheapest, CPU | embedding model; a GPU (graphics processor) helps |
c-TF-IDF: TF-IDF per cluster · SVD: singular value decomposition · CPU: central processor.
Stage 3 · which one for my corpus?
Which model, when
short documents?
tweets, titles, ticket subjects: too few shared words for counts
BERTopic
mixtures?
a document should mix several topics, with readable weights
LDA or NMF
cheap and stable?
no model download, an SVD-based start
NMF
No model wins everywhere: NMF beat LDA on the lesson's posts, LDA beat NMF in the lab. Measure against something the model never saw, and always read the topics.
Practice
To close the fundamental part
- Why is BERTopic's homogeneity of 0.927 not directly comparable with NMF's 0.646?
- Which model's topics changed the most with the seed, and which one changed even their number?
- For 50,000 product-review titles of eight words each, which model would you start with?
Next: the summary and the fundamental lab on the overview page (steps 1 to 7).