Machine Learning · Insper · E3.2

NMF

Advanced approach

How many topics, and are they any good?

The path of this part

Two scoresperplexity; NPMI by hand
Coherence measuredLDA against NMF, coherence and diversity
Choose ka scan from 2 to 20, then read the topics

Each stage answers one question: with no labels, what can score a topic? → which model has the better topics? → how many topics, then?

Stage 1 · with no labels, what can score a topic?

How many topics, and are they any good?

LDA (latent Dirichlet allocation) and NMF (non-negative matrix factorization) need \(k\) in advance, and there are no labels to cross-validate against.

perplexity

how surprised LDA is by held-out posts; lower is better; only for probabilistic models

coherence

do a topic's top words appear together in the documents?

\[ \text{NPMI}(a, b) = \frac{\log \dfrac{P(a, b)}{P(a)\,P(b)}}{-\log P(a, b)} \in [-1, 1] \]

NPMI: normalized pointwise mutual information. \(P(a)\): share of documents containing word \(a\); \(P(a, b)\): share containing both. 0 = unrelated, 1 = always together, −1 = never.

NPMI by hand, step by step

2,155 posts; 304 contain god, 159 jesus, 101 both. Predict. If the two words were unrelated, about how many posts would contain both?

Stage 2 · which model has the better topics?

Coherence and diversity, measured

Coherence of a topic: mean NPMI over all pairs of its top-10 words; of a model: the mean over its topics.
Diversity: share of distinct words among the top-10 words of all topics (1.0 = no topic shares a top word).

4 topics on the posts	coherence (NPMI)	diversity
LDA	0.224	0.85
NMF	0.369	1.00

Both scores agree with the eye: LDA repeats space, like, think across topics; NMF's topics share no top word.

Stage 3 · how many topics, then?

Choosing \(k\)

Predict. Held-out perplexity from \(k = 2\) to 20: will it point to 4, the number of forums?

Left: LDA's held-out perplexity rises steadily with k. Right: coherence; LDA peaks at k = 4, NMF climbs to k = 4 and stays on a plateau

Perplexity only rises: read literally, \(k = 2\); and it does not track what people find interpretable (Chang et al., 2009).

Coherence: LDA peaks at \(k = 4\) within a wide seed band; NMF climbs to 4, then a plateau.

Scan a range of \(k\), look for a knee, then read the topics at two or three values. Four forums or twelve sub-themes is a choice of granularity.

Practice · at home

Your turn (advanced)

A pair that never meets: the NPMI of hockey and jesus. Near −1, 0 or 1?
Coherence of BERTopic's words: does the score agree with how readable they look?
Your own scan: NMF coherence at \(k\) = 3, 4, 6, 8; read the topics at 8.

Next: the advanced slides of E3.3 BERTopic.