Machine Learning · Insper · E3

Topic Modeling & BERTopic

Advanced approach

Can the topics be trusted?

The path of this part

  1. Under LDA's hoodknown topics recovered; too few words
  2. How many topicsperplexity, NPMI coherence, the choice of k
  3. Name and renamec-TF-IDF on the posts; tuned topic words
  4. The pitfallsstop words, outliers, cliffs, seeds, language
  5. Use and checksearch, new posts, a fair comparison, outliers
Each stage answers one question: does fitting find the true topics? → how many topics, and who judges them? → where do the names come from? → which mistakes make a topic model lie? → what is a fitted model worth?

What each page adds

E3.1 LDA (latent Dirichlet allocation)

a corpus written by the story itself: the topics come back within 0.01; on 53 words, 1 seed in 20 finds them

E3.2 NMF (non-negative matrix factorization)

perplexity only rises with \(k\); coherence by NPMI (normalized pointwise mutual information), by hand; read the topics

E3.3 BERTopic

the 24 names reproduced by hand; update_topics renames without re-clustering

E3.4 Comparing

stop words, outliers, a cliff in min_cluster_size, a millionth of noise, the language

Stage 5 · what is a fitted model worth?

Step 8: a topic model as a tool

  • find_topics('car engine'): search the topics by meaning, with the query's embedding.
  • get_representative_docs(t): a few typical posts of a topic.
  • transform(new_docs): embed, reduce and cluster new documents with the fitted model, no refit.
Predict. The test part of the same five forums, 1,729 posts the model never saw: will the topics agree with their forums about as well as on the training posts?

Step 9: is the comparison fair?

ARI (adjusted Rand index) with the forums: 1 = the same partition, 0 = chance.

LDA

5 topics, places every post

NMF

5 topics, places every post

BERTopic

7 topics, 59 outliers left out

Predict. Score LDA and NMF only on the posts BERTopic kept. Do they catch up with BERTopic once the hard posts are gone?

Step 10: what to do with the outliers

  • The 59 outliers come mostly from medicine and the Middle East: cross-posted threads, replies arguing about replies.
  • reduce_outliers(strategy='embeddings') gives each one the topic of the closest centroid; update_topics recomputes the words.
Predict. With every post in a topic, does the agreement with the forums (ARI) go up or down? And the purity of the topics (homogeneity)?
Advanced lab · about 30 min, in pairs

Use, compare and clean the topics: steps 8 to 10

  • Same notebook, same session: the steps use docs, forum, E, topic_model from steps 1 to 7.
  • Step 8: search, typical posts, new sentences. Step 9: the three models against the forums; reduce_topics. Step 10: the outliers.
  • Simulations: next week's posts (step 8) and a fair comparison (step 9).

Pace: step 8 and the first simulation in 12 min; step 9 and the second simulation in 10 min; step 10 in 8 min.