E3.1 LDA

LDA (Latent Dirichlet Allocation) takes the first answer of What is a topic?: a topic is a distribution over words, and a document is a mixture of topics. This page tells its story by hand, shows what the Dirichlet prior controls and runs it on 2,155 real forum posts; then it checks, on a synthetic corpus, that fitting recovers the true topics, and watches it fail on the eleven hand-made sentences. Every number comes from the code next to it.

import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer

docs_toy = [
    "the team won the game with a late goal",                 # sports
    "fans cheered when the team scored a goal",
    "the coach said the game was a fair win for the team",
    "the rocket launch put a satellite in orbit",             # space
    "nasa delayed the rocket launch to the moon",
    "the satellite reached orbit around the moon",
    "bake the bread in a hot oven",                           # cooking
    "knead the dough and bake the bread slowly",
    "fresh bread from the oven smells great",
    "the team watched the rocket launch before the game",     # sports + space
    "astronauts ate fresh bread on the moon",                 # space + cooking
]
cv_toy = CountVectorizer(stop_words='english')
X_toy = cv_toy.fit_transform(docs_toy)
words_toy = cv_toy.get_feature_names_out()

How this page is organized

Like every page of the lesson, this one has two approaches, each with its own slides. The fundamental approach is part of the lesson's fundamental path, which the lesson quiz covers and the quiz at the end of this page reviews; the advanced one builds on it.

Fundamental approach

One question drives this part: what story could have written these words? LDA answers with a random process that draws a topic for every word slot, and fitting LDA means running that story backwards. This part tells the story by hand, shows what its prior controls, and runs it on real posts.

This part follows one path in three stages, drawn below: 1. tell the story; 2. the Dirichlet prior; 3. fit real posts. Each stage links to its section, and in the slides a strip at the bottom shows which stage you are in.

Fundamental approach slides

Click the slides and use the ← → arrows to navigate; F opens full screen and Esc shows every slide. Open the slides in a new tab

1. The generative story

Stage 1. Latent Dirichlet Allocation (Blei, Ng & Jordan, 2003) imagines that each document was written by a random process:

1. each topic \(k\) is a probability distribution \(\varphi_k\) over the vocabulary;
2. each document \(d\) has its own mixture of topics \(\theta_d\), for example \((0.8, 0.2)\);
3. for every word slot of the document, draw a topic \(z\) from \(\theta_d\), then draw the word from \(\varphi_z\).

It is a story of how the words got there, not a claim that people write this way. Word order plays no part in it, which is why LDA only needs bag-of-words counts. The story fixes the probability of each word in document \(d\):

By hand, with two topics over six words and a document that is 80% sports: \(p(\textit{team}) = 0.8 \times 0.40 + 0.2 \times 0.05 = 0.33\) and \(p(\textit{rocket}) = 0.8 \times 0.05 + 0.2 \times 0.40 = 0.12\). The code does the same for every word and then writes a 12-word document by following the story:

vocab = np.array(['team', 'goal', 'game', 'rocket', 'orbit', 'moon'])
phi = np.array([[0.40, 0.30, 0.20, 0.05, 0.00, 0.05],     # topic 0: "sports"
                [0.05, 0.00, 0.05, 0.40, 0.30, 0.20]])    # topic 1: "space"
theta = np.array([0.8, 0.2])                              # this document: 80% sports, 20% space

p_word = theta @ phi                                      # p(w|d) = sum over k of theta_k * phi_kw
print('  '.join(f"{w} {p:.2f}" for w, p in zip(vocab, p_word)))

rng = np.random.default_rng(0)
z = rng.choice(2, size=12, p=theta)                       # a topic for every word slot...
words = [rng.choice(vocab, p=phi[k]) for k in z]          # ...then a word drawn from that topic
print(z)
print(' '.join(words))

team 0.33  goal 0.24  game 0.17  rocket 0.12  orbit 0.06  moon 0.08
[0 0 0 0 1 1 0 0 0 1 1 0]
game team game team moon orbit team goal team rocket orbit goal

Four of the twelve slots drew topic 1, and those four slots are exactly where moon, orbit, rocket, orbit appear. The document reads as mostly sports with a bit of space, as its \(\theta\) says (figure script).

2. The Dirichlet prior: how mixed is a document?

Stage 2. The "Dirichlet" in the name is the distribution the mixtures \(\theta_d\) are drawn from. Its parameter \(\alpha\) controls how mixed a typical document is:

rng = np.random.default_rng(1)
for alpha in [0.1, 1.0, 10.0]:
    thetas = rng.dirichlet([alpha] * 3, size=2000)        # topic mixtures of 2000 documents
    print(f"alpha={alpha:<4}  one document: {thetas[0].round(2)}   "
          f"average largest share: {thetas.max(axis=1).mean():.2f}")

alpha=0.1   one document: [0. 1. 0.]   average largest share: 0.89
alpha=1.0   one document: [0.01 0.65 0.34]   average largest share: 0.62
alpha=10.0  one document: [0.18 0.46 0.36]   average largest share: 0.42

With \(\alpha = 0.1\) a document gives, on average, 89% of its words to a single topic; with \(\alpha = 10\) its largest topic gets only 42%, barely more than the \(1/3\) of a perfect mix (figure script). In scikit-learn, \(\alpha\) is the doc_topic_prior of LatentDirichletAllocation, \(1/k\) by default: smaller than 1, so LDA expects documents that concentrate on a few topics.

Try it below: drag the document's mixture \(\theta\) inside the triangle and watch the twelve word slots change color one by one; then set \(\alpha\) to 0.1 and to 10 and press draw θ from Dirichlet(α) a few times. The three topics use the \(\varphi\) of the synthetic corpus of the advanced approach (0.35, 0.30 and 0.25 on their own three words).

With a small \(\alpha\) almost every document you draw comes out in a single color; with a large \(\alpha\) every document mixes the three topics. The bars are \(p(w \mid d)\): the twelve words wobble around them.

3. LDA on real posts

Stage 3. Fitting LDA runs the story backwards: from the observed words alone, it estimates the topics \(\varphi\) and the mixtures \(\theta\). In scikit-learn, LatentDirichletAllocation(n_components=k).fit(X) does it on a matrix of counts; each row of components_, divided by its sum, is an estimated topic \(\varphi_k\), and transform(X) gives each document's \(\theta_d\). The advanced approach checks on a corpus with known topics that the estimates are right.

The real corpus: posts from four forums of the classic 20 Newsgroups collection, with headers, quoted replies and most signatures removed, and the posts shorter than 20 words dropped. The forum of each post is kept aside, in y, only to check the topics afterwards:

from sklearn.datasets import fetch_20newsgroups

forums = ['comp.graphics', 'rec.sport.hockey', 'sci.space', 'soc.religion.christian']
ng = fetch_20newsgroups(subset='train', categories=forums, random_state=0,
                        remove=('headers', 'footers', 'quotes'))   # keep only the body text
long_enough = np.array([len(text.split()) >= 20 for text in ng.data])
docs = [text for text, keep in zip(ng.data, long_enough) if keep]
y = ng.target[long_enough]                                         # the forum: used only to check
print(len(ng.data), '->', len(docs), 'posts with at least 20 words')
print(pd.Series(np.array(forums)[y]).value_counts())

2376 -> 2155 posts with at least 20 words
soc.religion.christian    565
sci.space                 535
rec.sport.hockey          534
comp.graphics             521
Name: count, dtype: int64

from sklearn.decomposition import LatentDirichletAllocation
from sklearn.metrics import adjusted_rand_score

def top_words(components, vocab, n=4):
    return [' '.join(vocab[row.argsort()[::-1][:n]]) for row in components]

settings = dict(stop_words='english', min_df=5, max_df=0.5,
                token_pattern=r'(?u)\b[a-zA-Z]{3,}\b')             # words of 3+ letters, no numbers
cv = CountVectorizer(**settings)
X = cv.fit_transform(docs)
vocab = cv.get_feature_names_out()
print(X.shape)

lda = LatentDirichletAllocation(n_components=4, random_state=0).fit(X)
print(*top_words(lda.components_, vocab, 8), sep='\n')
lda_label = lda.transform(X).argmax(axis=1)       # each post goes to its largest topic
print(f"ARI with the forums: {adjusted_rand_score(y, lda_label):.3f}")

(2155, 5725)
jpeg don pts like think play period gif
god space people think just jesus time like
team game hockey season games nhl play year
image edu data graphics space available software program
ARI with the forums: 0.456

The vocabulary keeps words of three letters or more that appear in at least 5 posts and in at most half of them. top_words lists the heaviest words of each row of components_. Hockey comes out clean (team game hockey season nhl). The other three topics are blurred: religion shares its topic with space, graphics with space too, and one topic is made of words like jpeg, pts, period, gif, file formats mixed with hockey scoring tables. To compare the topics with the forums, each post is sent to its largest topic and the result is compared with the forums by the adjusted Rand index (1 = identical partitions, 0 = no better than chance): 0.456.

4. Summary: LDA in four lines

1. A topic is a distribution \(\varphi_k\) over words, and a document a mixture \(\theta_d\) of topics; every word slot draws a topic, then a word.
2. The probability of a word in a document is the mixture of the topics' probabilities: \(p(w \mid d) = \sum_k \theta_{d,k}\, \varphi_{k,w}\).
3. The Dirichlet prior \(\alpha\) says how mixed a typical document is: small \(\alpha\), one topic per document; large \(\alpha\), everything mixed. scikit-learn's default, \(1/k\), favors concentrated documents.
4. Fitting runs the story backwards on bag-of-words counts: components_ holds the topics, transform the mixtures. Read the topics by their top words and check them against anything the model did not see.

In the lab. Step 2 fits LDA on five other forums.

To close the fundamental part. In one sentence each: (1) why does LDA need only word counts, and not word order? (2) With the two topics of section 1, which of the six words could never appear in a document that is 100% space? (3) What would the documents of a corpus look like if its \(\alpha\) were 10?

Your turn (fundamental)

1. Another mixture. Rerun the block of section 1 with \(\theta = (0.3, 0.7)\) and another seed. Which words become the most likely, and how many of the 12 slots draw the space topic?
2. More topics. Refit LDA on the posts with n_components=8. Do the blurred topics separate? Does the ARI with the four forums go up or down, and why should it not be read as a score of quality here?

Advanced approach

The fundamental approach told LDA's story and ran it on real posts, but a blurred topic leaves a doubt: is the model wrong, or is the data hard? This part settles it in two experiments. First, a corpus generated by the story itself, so that the true topics are known; then the eleven hand-made sentences, where the counts carry too little evidence. It ends with the knobs scikit-learn exposes. The code blocks continue those of sections 1 to 3.

This part follows one path in three stages, drawn below: 1. recover known topics; 2. too few words; 3. the knobs. Each stage links to its section, and in the slides a strip at the bottom shows which stage you are in.

Advanced approach slides

Click the slides and use the ← → arrows to navigate; F opens full screen and Esc shows every slide. Open the slides in a new tab

5. Fitting LDA: running the story backwards

Stage 1. Fitting LDA inverts the story: from the observed words alone, estimate the topics \(\varphi\) and the mixtures \(\theta\). scikit-learn does it with variational inference, an iterative optimization that starts from a random point. To see whether it works, generate a corpus with the story itself, so that the true topics are known, and ask LDA to find them:

from sklearn.decomposition import LatentDirichletAllocation

vocab9 = ['team', 'goal', 'game', 'rocket', 'orbit', 'moon', 'bread', 'oven', 'bake']
phi_true = np.full((3, 9), 0.10 / 6)                      # a little of every word...
for k in range(3):
    phi_true[k, 3 * k:3 * k + 3] = [0.35, 0.30, 0.25]     # ...and 90% on its own three words
rng = np.random.default_rng(0)
theta_true = rng.dirichlet([0.3] * 3, size=300)           # 300 documents
X_syn = np.array([rng.multinomial(30, t @ phi_true) for t in theta_true])   # 30 words each

lda_syn = LatentDirichletAllocation(n_components=3, random_state=0).fit(X_syn)
phi_hat = lda_syn.components_ / lda_syn.components_.sum(axis=1, keepdims=True)
order = [int(np.argmax(phi_hat[:, 3 * k])) for k in range(3)]   # line the topics up with the truth
print(pd.DataFrame(phi_hat[order], columns=vocab9).round(2))
theta_hat = lda_syn.transform(X_syn)[:, order]
print("correlation of true and estimated theta, per topic:",
      [round(float(np.corrcoef(theta_true[:, k], theta_hat[:, k])[0, 1]), 3) for k in range(3)])

   team  goal  game  rocket  orbit  moon  bread  oven  bake
0  0.35  0.31  0.25    0.02   0.01  0.02   0.01  0.02  0.02
1  0.01  0.01  0.02    0.36   0.31  0.26   0.01  0.01  0.01
2  0.01  0.01  0.01    0.01   0.01  0.02   0.34  0.32  0.26
correlation of true and estimated theta, per topic: [0.976, 0.974, 0.973]

components_ holds one row per topic; divided by its sum, a row is the estimated \(\varphi_k\). The true topics put 0.35, 0.30 and 0.25 on their own three words, and LDA recovers them within 0.01. The mixtures come back too: the estimated share of each topic correlates 0.97 with the true one. The topics come out in an arbitrary order (topic "2" of one run can be topic "0" of another), which is why the code lines them up with the truth first.

6. Too few words: the hand-made sentences

Stage 2. That corpus had 9,000 words. The hand-written one has 53:

lda_toy = LatentDirichletAllocation(n_components=3, random_state=0).fit(X_toy)
print(top_words(lda_toy.components_, words_toy))
print(lda_toy.transform(X_toy).round(2)[[0, 3, 6, 9]])

def clean(labels):            # docs 0-2, 3-5, 6-8 each in one topic, three different topics
    return all(len(set(labels[i:i + 3])) == 1 for i in (0, 3, 6)) and len(set(labels[:9])) == 3

n_clean = sum(clean(LatentDirichletAllocation(n_components=3, random_state=s)
                    .fit(X_toy).transform(X_toy).argmax(axis=1)) for s in range(20))
print("seeds with a clean split:", n_clean, "of 20")

['bread fresh game oven', 'team goal game orbit', 'rocket launch bread delayed']
[[0.06 0.88 0.06]
 [0.07 0.08 0.86]
 [0.86 0.07 0.07]
 [0.06 0.07 0.87]]
seeds with a clean split: 1 of 20

With seed 0, LDA mixes the themes: bread and game share a topic, orbit sits with team. Each document is still given confidently to one topic (0.86 to 0.88), but the topics are not the three themes. Over 20 seeds, only one separates the nine single-theme sentences cleanly. Nothing is wrong with the code: with 53 words the counts carry too little evidence, and the optimization settles wherever its random start leads. LDA needs many documents and enough words in each one; on the synthetic corpus it had both. The NMF page runs on the same 11 sentences.

7. The knobs: two priors and a seed

Stage 3. A second Dirichlet, with parameter \(\eta\), does for the words of each topic what \(\alpha\) does for the topics of each document: a small \(\eta\) gives topics that concentrate on a few words. In scikit-learn they are doc_topic_prior (\(\alpha\)) and topic_word_prior (\(\eta\)) of LatentDirichletAllocation, both \(1/k\) by default, which favors documents and topics that concentrate on a few items. Two more choices are not priors but change the result just as much: the number of topics \(k\), fixed in advance (the NMF page discusses how to choose it), and the random start, random_state, which section 6 showed can decide everything when the evidence is thin.

Your turn (advanced)

1. Less data, same story. Rerun section 5 with 30 documents instead of 300, then with 5 words per document instead of 30. How far are the estimated \(\varphi\) from the truth in each case? Which matters more, many documents or long ones?
2. A wrong prior. Generate the synthetic corpus with rng.dirichlet([3.0] * 3, ...) (mixed documents) and fit LDA with doc_topic_prior=0.05. Does LDA still recover the topics? And the mixtures?
3. Seeds on the posts. Fit the LDA of section 3 with random_state 0 to 4 and compute the ARI between the runs. Is the 53-word lottery of section 6 also a problem with 2,155 posts?

References

• Blei, D. M.; Ng, A. Y.; Jordan, M. I. "Latent Dirichlet Allocation." JMLR 3 (2003): 993–1022 — the generative story and its variational inference. full text

Quiz

The quiz reviews the fundamental approach of this page; the lesson quiz covers the five pages.

Next: E3.2 NMF, the same topics by matrix factorization.