Skip to content

U5.1 LDA

LDA (Latent Dirichlet Allocation) takes the first answer of What is a topic?: a topic is a distribution over words, and a document is a mixture of topics. This page tells its story by hand, shows what the Dirichlet prior controls, checks on a synthetic corpus that fitting recovers the true topics, and then runs it on the eleven hand-made sentences and on 2,155 real forum posts. Every number comes from the code next to it.

  1. What is a topic?words that go together, or documents that go together
  2. LDALatent Dirichlet Allocation: a story of how words get written
  3. NMFnon-negative matrix factorization: a document as a sum of parts
  4. BERTopicembed, reduce, cluster, describe
  5. Compare and choosethe three side by side; the pitfalls
Setup: what this page needs from the previous ones

This page continues the overview. If you start here, run this block first: it rebuilds the hand-made corpus.

import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer

docs_toy = [
    "the team won the game with a late goal",                 # sports
    "fans cheered when the team scored a goal",
    "the coach said the game was a fair win for the team",
    "the rocket launch put a satellite in orbit",             # space
    "nasa delayed the rocket launch to the moon",
    "the satellite reached orbit around the moon",
    "bake the bread in a hot oven",                           # cooking
    "knead the dough and bake the bread slowly",
    "fresh bread from the oven smells great",
    "the team watched the rocket launch before the game",     # sports + space
    "astronauts ate fresh bread on the moon",                 # space + cooking
]
cv_toy = CountVectorizer(stop_words='english')
X_toy = cv_toy.fit_transform(docs_toy)
words_toy = cv_toy.get_feature_names_out()

The generative story

Stage 2. Latent Dirichlet Allocation (Blei, Ng & Jordan, 2003) imagines that each document was written by a random process:

  1. each topic \(k\) is a probability distribution \(\varphi_k\) over the vocabulary;
  2. each document \(d\) has its own mixture of topics \(\theta_d\), for example \((0.8, 0.2)\);
  3. for every word slot of the document, draw a topic \(z\) from \(\theta_d\), then draw the word from \(\varphi_z\).

It is a story of how the words got there, not a claim that people write this way. Word order plays no part in it, which is why LDA only needs bag-of-words counts. The story fixes the probability of each word in document \(d\):

\[ p(w \mid d) = \sum_k \theta_{d,k}\, \varphi_{k,w} \]

By hand, with two topics over six words and a document that is 80% sports: \(p(\textit{team}) = 0.8 \times 0.40 + 0.2 \times 0.05 = 0.33\) and \(p(\textit{rocket}) = 0.8 \times 0.05 + 0.2 \times 0.40 = 0.12\). The code does the same for every word and then writes a 12-word document by following the story:

vocab = np.array(['team', 'goal', 'game', 'rocket', 'orbit', 'moon'])
phi = np.array([[0.40, 0.30, 0.20, 0.05, 0.00, 0.05],     # topic 0: "sports"
                [0.05, 0.00, 0.05, 0.40, 0.30, 0.20]])    # topic 1: "space"
theta = np.array([0.8, 0.2])                              # this document: 80% sports, 20% space

p_word = theta @ phi                                      # p(w|d) = sum over k of theta_k * phi_kw
print('  '.join(f"{w} {p:.2f}" for w, p in zip(vocab, p_word)))

rng = np.random.default_rng(0)
z = rng.choice(2, size=12, p=theta)                       # a topic for every word slot...
words = [rng.choice(vocab, p=phi[k]) for k in z]          # ...then a word drawn from that topic
print(z)
print(' '.join(words))

team 0.33  goal 0.24  game 0.17  rocket 0.12  orbit 0.06  moon 0.08
[0 0 0 0 1 1 0 0 0 1 1 0]
game team game team moon orbit team goal team rocket orbit goal

Two topics as bar charts over six words (sports: team 0.40, goal 0.30, game 0.20; space: rocket 0.40, orbit 0.30, moon 0.20) and the document's word distribution, 0.8 times the first plus 0.2 times the second: team 0.33, goal 0.24, game 0.17, rocket 0.12, orbit 0.06, moon 0.08

Four of the twelve slots drew topic 1, and those four slots are exactly where moon, orbit, rocket, orbit appear. The document reads as mostly sports with a bit of space, as its \(\theta\) says (figure script).

The Dirichlet prior: how mixed is a document?

The "Dirichlet" in the name is the distribution the mixtures \(\theta_d\) are drawn from. Its parameter \(\alpha\) controls how mixed a typical document is:

rng = np.random.default_rng(1)
for alpha in [0.1, 1.0, 10.0]:
    thetas = rng.dirichlet([alpha] * 3, size=2000)        # topic mixtures of 2000 documents
    print(f"alpha={alpha:<4}  one document: {thetas[0].round(2)}   "
          f"average largest share: {thetas.max(axis=1).mean():.2f}")

alpha=0.1   one document: [0. 1. 0.]   average largest share: 0.89
alpha=1.0   one document: [0.01 0.65 0.34]   average largest share: 0.62
alpha=10.0  one document: [0.18 0.46 0.36]   average largest share: 0.42

Mixtures of 400 documents over three topics drawn inside a triangle whose corners are the pure topics: with alpha 0.1 the points pile up on the corners (largest share 0.89 on average), with alpha 1 they spread over the triangle (0.62), with alpha 10 they crowd the center (0.42)

With \(\alpha = 0.1\) a document gives, on average, 89% of its words to a single topic; with \(\alpha = 10\) its largest topic gets only 42%, barely more than the \(1/3\) of a perfect mix (figure script). A second Dirichlet, with parameter \(\eta\), does the same for the words of each topic. In scikit-learn they are doc_topic_prior and topic_word_prior of LatentDirichletAllocation, both \(1/k\) by default, which favors documents and topics that concentrate on a few items.

Try it below: drag the document's mixture \(\theta\) inside the triangle and watch the twelve word slots change color one by one; then set \(\alpha\) to 0.1 and to 10 and press draw θ from Dirichlet(α) a few times. The three topics use the \(\varphi\) of the synthetic corpus of the next section (0.35, 0.30 and 0.25 on their own three words).

With a small \(\alpha\) almost every document you draw comes out in a single color; with a large \(\alpha\) every document mixes the three topics. The bars are \(p(w \mid d)\): the twelve words wobble around them.

Fitting LDA: running the story backwards

Fitting LDA inverts the story: from the observed words alone, estimate the topics \(\varphi\) and the mixtures \(\theta\). scikit-learn does it with variational inference, an iterative optimization that starts from a random point. To see whether it works, generate a corpus with the story itself, so that the true topics are known, and ask LDA to find them:

from sklearn.decomposition import LatentDirichletAllocation

vocab9 = ['team', 'goal', 'game', 'rocket', 'orbit', 'moon', 'bread', 'oven', 'bake']
phi_true = np.full((3, 9), 0.10 / 6)                      # a little of every word...
for k in range(3):
    phi_true[k, 3 * k:3 * k + 3] = [0.35, 0.30, 0.25]     # ...and 90% on its own three words
rng = np.random.default_rng(0)
theta_true = rng.dirichlet([0.3] * 3, size=300)           # 300 documents
X_syn = np.array([rng.multinomial(30, t @ phi_true) for t in theta_true])   # 30 words each

lda_syn = LatentDirichletAllocation(n_components=3, random_state=0).fit(X_syn)
phi_hat = lda_syn.components_ / lda_syn.components_.sum(axis=1, keepdims=True)
order = [int(np.argmax(phi_hat[:, 3 * k])) for k in range(3)]   # line the topics up with the truth
print(pd.DataFrame(phi_hat[order], columns=vocab9).round(2))
theta_hat = lda_syn.transform(X_syn)[:, order]
print("correlation of true and estimated theta, per topic:",
      [round(float(np.corrcoef(theta_true[:, k], theta_hat[:, k])[0, 1]), 3) for k in range(3)])

   team  goal  game  rocket  orbit  moon  bread  oven  bake
0  0.35  0.31  0.25    0.02   0.01  0.02   0.01  0.02  0.02
1  0.01  0.01  0.02    0.36   0.31  0.26   0.01  0.01  0.01
2  0.01  0.01  0.01    0.01   0.01  0.02   0.34  0.32  0.26
correlation of true and estimated theta, per topic: [0.976, 0.974, 0.973]

components_ holds one row per topic; divided by its sum, a row is the estimated \(\varphi_k\). The true topics put 0.35, 0.30 and 0.25 on their own three words, and LDA recovers them within 0.01. The mixtures come back too: the estimated share of each topic correlates 0.97 with the true one. The topics come out in an arbitrary order (topic "2" of one run can be topic "0" of another), which is why the code lines them up with the truth first.

That corpus had 9,000 words. The hand-written one has 53:

def top_words(components, vocab, n=4):
    return [' '.join(vocab[row.argsort()[::-1][:n]]) for row in components]

lda_toy = LatentDirichletAllocation(n_components=3, random_state=0).fit(X_toy)
print(top_words(lda_toy.components_, words_toy))
print(lda_toy.transform(X_toy).round(2)[[0, 3, 6, 9]])

def clean(labels):            # docs 0-2, 3-5, 6-8 each in one topic, three different topics
    return all(len(set(labels[i:i + 3])) == 1 for i in (0, 3, 6)) and len(set(labels[:9])) == 3

n_clean = sum(clean(LatentDirichletAllocation(n_components=3, random_state=s)
                    .fit(X_toy).transform(X_toy).argmax(axis=1)) for s in range(20))
print("seeds with a clean split:", n_clean, "of 20")

['bread fresh game oven', 'team goal game orbit', 'rocket launch bread delayed']
[[0.06 0.88 0.06]
 [0.07 0.08 0.86]
 [0.86 0.07 0.07]
 [0.06 0.07 0.87]]
seeds with a clean split: 1 of 20

With seed 0, LDA mixes the themes: bread and game share a topic, orbit sits with team. Each document is still given confidently to one topic (0.86 to 0.88), but the topics are not the three themes. Over 20 seeds, only one separates the nine single-theme sentences cleanly. Nothing is wrong with the code: with 53 words the counts carry too little evidence, and the optimization settles wherever its random start leads. LDA needs many documents and enough words in each one; on the synthetic corpus it had both.

LDA on real posts

The real corpus: posts from four forums of the classic 20 Newsgroups collection, with headers, quoted replies and most signatures removed, and the posts shorter than 20 words dropped. The forum of each post is kept aside, in y, only to check the topics afterwards:

from sklearn.datasets import fetch_20newsgroups

forums = ['comp.graphics', 'rec.sport.hockey', 'sci.space', 'soc.religion.christian']
ng = fetch_20newsgroups(subset='train', categories=forums, random_state=0,
                        remove=('headers', 'footers', 'quotes'))   # keep only the body text
long_enough = np.array([len(text.split()) >= 20 for text in ng.data])
docs = [text for text, keep in zip(ng.data, long_enough) if keep]
y = ng.target[long_enough]                                         # the forum: used only to check
print(len(ng.data), '->', len(docs), 'posts with at least 20 words')
print(pd.Series(np.array(forums)[y]).value_counts())

2376 -> 2155 posts with at least 20 words
soc.religion.christian    565
sci.space                 535
rec.sport.hockey          534
comp.graphics             521
Name: count, dtype: int64

from sklearn.metrics import adjusted_rand_score

settings = dict(stop_words='english', min_df=5, max_df=0.5,
                token_pattern=r'(?u)\b[a-zA-Z]{3,}\b')             # words of 3+ letters, no numbers
cv = CountVectorizer(**settings)
X = cv.fit_transform(docs)
vocab = cv.get_feature_names_out()
print(X.shape)

lda = LatentDirichletAllocation(n_components=4, random_state=0).fit(X)
print(*top_words(lda.components_, vocab, 8), sep='\n')
lda_label = lda.transform(X).argmax(axis=1)       # each post goes to its largest topic
print(f"ARI with the forums: {adjusted_rand_score(y, lda_label):.3f}")

(2155, 5725)
jpeg don pts like think play period gif
god space people think just jesus time like
team game hockey season games nhl play year
image edu data graphics space available software program
ARI with the forums: 0.456

The vocabulary keeps words of three letters or more that appear in at least 5 posts and in at most half of them. Hockey comes out clean (team game hockey season nhl). The other three topics are blurred: religion shares its topic with space, graphics with space too, and one topic is made of words like jpeg, pts, period, gif, file formats mixed with hockey scoring tables. To compare the topics with the forums, each post is sent to its largest topic and the result is compared with the forums by the adjusted Rand index (1 = identical partitions, 0 = no better than chance): 0.456.

References

  • Blei, D. M.; Ng, A. Y.; Jordan, M. I. "Latent Dirichlet Allocation." JMLR 3 (2003): 993–1022 — the generative story and its variational inference. full text

Next: U5.2 NMF, the same topics by matrix factorization.