Machine Learning · Insper · E3.1
LDA
Advanced approach
Does fitting recover the topics, and when does it fail?
The path of this part
- Recover known topicsa synthetic corpus, written by the story
- Too few words53 words, 20 seeds
- The knobsα, η and the random start
Each stage answers one question: does LDA (latent Dirichlet allocation) find topics that are known? → what happens with too little evidence? → what can be set, and what does it change?
Stage 1 · does LDA find topics that are known?
A corpus written by the story itself
- 3 topics over 9 words: each puts 0.35, 0.30, 0.25 on its own three words and \(0.10/6\) on each of the others.
- 300 documents, mixtures \(\theta_d \sim \text{Dirichlet}(0.3)\), 30 words each: 9,000 words.
- Fit
LatentDirichletAllocation(n_components=3) on the counts alone.
Predict. Will the estimated topics be close to the true ones? Will they come out in the same order?
The topics come back
| team | goal | game | rocket | orbit | moon | bread | oven | bake |
|---|
| sports | 0.35 | 0.31 | 0.25 | 0.02 | 0.01 | 0.02 | 0.01 | 0.02 | 0.02 |
| space | 0.01 | 0.01 | 0.02 | 0.36 | 0.31 | 0.26 | 0.01 | 0.01 | 0.01 |
| cooking | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | 0.02 | 0.34 | 0.32 | 0.26 |
- Each row of
components_, divided by its sum, against the truth: within 0.01.
- Estimated against true mixtures \(\theta\): correlation 0.97 for every topic.
The topics come out in an arbitrary order: the code lines them up with the truth first. Topic "2" of one run can be topic "0" of another.
Stage 2 · what happens with too little evidence?
And on the 53 hand-written words?
Predict. Three topics on the 11 hand-made sentences: will LDA separate sports, space and cooking?
- Seed 0: bread fresh game oven · team goal game orbit · rocket launch bread delayed: the themes mixed.
- Each document still goes confidently (0.86 to 0.88) to one topic.
- Over 20 seeds, only 1 separates the nine single-theme sentences cleanly.
Nothing is wrong with the code: 53 words carry too little evidence and the random start decides. LDA needs many documents with enough words each.
Stage 3 · what can be set, and what does it change?
The knobs: two priors and a seed
\(\alpha\): doc_topic_prior
how mixed a document is; small → few topics per document. Default \(1/k\).
\(\eta\): topic_word_prior
how spread a topic is over the words; small → few words per topic. Default \(1/k\).
\(k\): n_components
fixed in advance; the advanced part of E3.2 NMF discusses how to choose it
random_state
the random start of the optimization: it decided everything on 53 words
Practice · at home
Your turn (advanced)
- Less data, same story: 30 documents instead of 300, then 5 words each instead of 30. Which matters more?
- A wrong prior: mixed documents (\(\text{Dirichlet}(3)\)) fitted with
doc_topic_prior=0.05.
- Seeds on the posts: is the 53-word lottery a problem with 2,155 posts?
Next: the advanced slides of E3.2 NMF.