Machine Learning · Insper · E3.4
Comparing the Models
Advanced approach
The pitfalls that catch most topic-modeling projects
The path of this part
- Stop wordsjunk words and junk topics
- Outliersreassigning them is a trade-off
- Granularitya cliff, not a dial
- Reproducibilitya millionth of noise, 3 topics
- Languagea model for the corpus's language
Each stage answers one question: which words name nothing? → what to do with documents that belong nowhere? → how does the number of topics move? → will the topics come back tomorrow? → does the model speak the corpus's language?
Stage 1 · which words name nothing?
Pitfall: stop words in the representation
- Without a list, BERTopic names topics by the, of, to; even with the English list, LDA (latent Dirichlet allocation) kept don, think, like, just.
- A list in the wrong language does not help: topic 6 here is van, het, een, from a handful of posts in Dutch.
- Classic junk topics: signatures, e-mail footers, mailing-list templates: text repeated word for word forms very dense clusters.
The words that name a topic depend on the vectorizer: fix the list, then update_topics.
Stage 2 · what to do with documents that belong nowhere?
Pitfall: too many outliers
431 posts (20%) in topic −1. reduce_outliers gives each one the nearest topic, by its c-TF-IDF (TF-IDF per cluster) or by its embedding.
Predict. Measured on all posts, will the homogeneity stay at 0.927?
| strategy | outliers | homogeneity, all posts |
|---|
| c-tf-idf | 0 | 0.822 |
| embeddings | 0 | 0.882 |
The outliers were the ambiguous posts: forcing them in makes the topics less pure. A trade-off, not a free fix; threshold keeps the least similar ones out.
Stage 3 · how does the number of topics move?
Pitfall: granularity is a cliff, not a dial
Predict. min_cluster_size from 10 to 15: a few topics fewer, or a lot?
| min_cluster_size | topics | outliers |
|---|
| 5 | 96 | 28% |
| 10 | 24 | 20% |
| 15 | 5 | 0% |
| 25 | 3 | 0% |
HDBSCAN's "excess of mass" rule switches to the large parent clusters. Scan a few values and read the topics; to merge without re-clustering, reduce_topics.
Stage 4 · will the topics come back tomorrow?
Pitfall: reproducibility
Add to the embeddings a noise of about one millionth (the size of CPU-vs-GPU differences), keep every seed:
| noise, seed 0 | 24 topics, ARI with the original 0.908 |
| noise, seed 1 | 24 topics, ARI 0.910 |
| noise, seed 2 | 3 topics, ARI 0.514 |
ARI: adjusted Rand index (1 = same partition) · CPU, GPU: central and graphics processors.
min_cluster_size=10 sits at the edge of the cliff. Fixing seeds makes a run repeatable, not true: report topics that survive seeds and small perturbations, and save the fitted model.
Stage 5 · does the model speak the corpus's language?
Pitfall: the language of the embedding model
all-MiniLM-L6-v2 was trained on English.- For Portuguese or mixed corpora: a multilingual model such as
paraphrase-multilingual-MiniLM-L12-v2, or BERTopic(language='multilingual').
- And a Portuguese stop-word list in the vectorizer: the names come from it, not from the embeddings.
Practice · at home
Your turn (advanced)
- A threshold for the outliers:
reduce_outliers(..., threshold=0.3). How many stay outliers?
- The cliff, in finer steps: every
min_cluster_size from 10 to 15.
- Away from the cliff: the noise experiment with
min_cluster_size=15.
Next: the advanced lab on the overview page (steps 8 to 10).