Topic Modeling & BERTopic

Topic modeling answers a question that sounds simple: given thousands of documents, what are they about? Nobody labels the documents and nobody lists the themes in advance, so it is unsupervised learning, the text-world sibling of clustering. Typical uses: reading a year of customer reviews or support tickets without reading them, organizing a news archive, following what people discuss on social media, mapping a scientific literature.

This lesson builds three topic models, from the classical to the modern. LDA tells a probabilistic story about how words are written. NMF factorizes the TF-IDF matrix of the text representation lessons. BERTopic is almost entirely made of pieces you already know: sentence embeddings, UMAP, HDBSCAN and a variant of TF-IDF. Two corpora run through the whole lesson: eleven sentences written by hand, small enough to follow with a calculator, and 2,155 real posts from four internet forums, whose forum we know but hide from the models, so that we can check what they find. Every number below comes from the code shown next to it, and each figure is drawn by a short script that sits next to its page.

The lesson follows one path in five stages, drawn below: 1. what is a topic?, the two answers and a hand-made corpus; 2. LDA, a topic as a distribution over words, told as a probabilistic story; 3. NMF, the same answer with linear algebra; 4. BERTopic, a topic as a cluster of documents; 5. compare and choose, the three models measured side by side, with the pitfalls. Stage 1 is on this page and each of the other stages has its own page. Each stage links to its section or page, and in the slides a strip at the bottom shows which stage you are in.

Slides

Click the slides and use the ← → arrows to navigate; F goes full screen and Esc shows every slide. Open the slides in a new tab

What is a topic?

Stage 1. There are two answers, and the models of this lesson use one or the other:

• a topic is a distribution over words. "Sports" gives high probability to team, game, goal and almost none to oven. A document is a mixture of topics: 80% sports and 20% space, say. LDA works this way, and NMF too, with non-negative weights instead of probabilities;
• a topic is a cluster of documents. Each document belongs to one topic (or to none), and the topic's words are computed afterwards, to describe the cluster. BERTopic works this way.

Both start from documents turned into numbers. Here is the hand-made corpus, as the bag-of-words counts of the text representation lessons:

import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer

docs_toy = [
    "the team won the game with a late goal",                 # sports
    "fans cheered when the team scored a goal",
    "the coach said the game was a fair win for the team",
    "the rocket launch put a satellite in orbit",             # space
    "nasa delayed the rocket launch to the moon",
    "the satellite reached orbit around the moon",
    "bake the bread in a hot oven",                           # cooking
    "knead the dough and bake the bread slowly",
    "fresh bread from the oven smells great",
    "the team watched the rocket launch before the game",     # sports + space
    "astronauts ate fresh bread on the moon",                 # space + cooking
]
cv_toy = CountVectorizer(stop_words='english')
X_toy = cv_toy.fit_transform(docs_toy)
words_toy = cv_toy.get_feature_names_out()
print(X_toy.shape, '| word tokens kept:', X_toy.sum())
counts = pd.DataFrame(X_toy.toarray(), columns=words_toy)
print(counts[['team', 'game', 'goal', 'rocket', 'launch', 'moon', 'bread', 'bake']])

(11, 33) | word tokens kept: 53
    team  game  goal  rocket  launch  moon  bread  bake
0      1     1     1       0       0     0      0     0
1      1     0     1       0       0     0      0     0
2      1     1     0       0       0     0      0     0
3      0     0     0       1       1     0      0     0
4      0     0     0       1       1     1      0     0
5      0     0     0       0       0     1      0     0
6      0     0     0       0       0     0      1     1
7      0     0     0       0       0     0      1     1
8      0     0     0       0       0     0      1     0
9      1     1     0       1       1     0      0     0
10     0     0     0       0       0     1      1     0

After the English stop words (the, a, with, ...) are removed, 11 sentences keep 53 words drawn from a vocabulary of 33. Three themes are easy to see, and the last two sentences mix them. A topic model receives only this matrix, without the comments, and must find the themes.

The pages of this lesson

The lesson continues in four pages, meant to be read in order. Each page's code continues the previous one, and each page also opens with a setup block that rebuilds what it needs, so it can be run on its own.

The lab below repeats the whole lesson on five other forums, and the quiz closes the lesson.

Lab: topics in five forums, three ways

The lab repeats the lesson on a new corpus, five other forums of 20 Newsgroups, and compares the three models against the forums they came from. Steps 2 and 3 fit LDA and NMF; steps 4 to 7 build BERTopic step by step and then in one call; steps 8 to 10 inspect the topics, compare them with the forums and deal with the outliers.

Run the blocks in order, in one session. Every output below is what the code prints; on another machine the embeddings, and so BERTopic's numbers, can differ slightly. To work on it directly, download the lab notebook and open it in Jupyter or Colab (File → Upload notebook). In Colab, run !pip install bertopic in a cell first. The whole lab takes two to three minutes on a CPU, most of it in step 4.

Step 1 — load five forums

import numpy as np
import pandas as pd
from sklearn.datasets import fetch_20newsgroups

forums = ['comp.windows.x', 'rec.autos', 'rec.sport.baseball', 'sci.med', 'talk.politics.mideast']
ng = fetch_20newsgroups(subset='train', categories=forums, random_state=0,
                        remove=('headers', 'footers', 'quotes'))
keep = np.array([len(text.split()) >= 20 for text in ng.data])
docs = [text for text, k in zip(ng.data, keep) if k]
forum = np.array(ng.target_names)[ng.target[keep]]     # hidden from the models; used to check them
print(len(docs), 'posts')
print(pd.Series(forum).value_counts())

2613 posts
sci.med                  551
comp.windows.x           549
rec.autos                510
talk.politics.mideast    508
rec.sport.baseball       495
Name: count, dtype: int64

2,613 posts, between 495 and 551 per forum. The forum names are kept in forum for steps 2, 3, 6 and 9; none of the models sees them. talk.politics.mideast discusses several different conflicts, which will matter from step 6 on.

Step 2 — LDA

from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics import adjusted_rand_score

settings = dict(stop_words='english', min_df=5, max_df=0.5, token_pattern=r'(?u)\b[a-zA-Z]{3,}\b')
cv = CountVectorizer(**settings)
X = cv.fit_transform(docs)
vocab = cv.get_feature_names_out()

def top_words(components, n=8):
    return [' '.join(vocab[row.argsort()[::-1][:n]]) for row in components]

lda = LatentDirichletAllocation(n_components=5, random_state=0).fit(X)
lda_topic = lda.transform(X).argmax(axis=1)
print(*top_words(lda.components_), sep='\n')
print(f"ARI with the forums: {adjusted_rand_score(forum, lda_topic):.3f}")

car entry like don file just output use
year good think don game just team like
people armenian said armenians israel turkish jews know
window edu com server use available motif file
edu health medical use university research patients disease
ARI with the forums: 0.603

Five topics for five forums. The X Window System (window, server, motif), the Middle East and medicine are recognizable; baseball hides behind generic words (year good think don game), and the first topic mixes cars with software words (entry, output, file). Sending each post to its largest topic gives an ARI of 0.603 with the forums.

Step 3 — NMF

from sklearn.decomposition import NMF
from sklearn.feature_extraction.text import TfidfVectorizer

T = TfidfVectorizer(**settings).fit_transform(docs)     # same vocabulary as X
nmf = NMF(n_components=5, init='nndsvda', max_iter=500, random_state=0).fit(T)
nmf_topic = nmf.transform(T).argmax(axis=1)
print(*top_words(nmf.components_), sep='\n')
print(f"ARI with the forums: {adjusted_rand_score(forum, nmf_topic):.3f}")
print(pd.crosstab(nmf_topic, forum, rownames=['NMF topic'], colnames=['forum']).to_string())

car don just year good think like know
geb pitt cadre dsl chastity skepticism shameful intellect
window server motif use thanks application widget program
armenian armenians turkish people armenia turks turkey genocide
israel israeli jews arab arabs jewish peace lebanese
ARI with the forums: 0.333
forum      comp.windows.x  rec.autos  rec.sport.baseball  sci.med  talk.politics.mideast
NMF topic
0                      12        460                 464      394                     66
1                       0          2                   0       79                      0
2                     537         41                  22       70                     14
3                       0          1                   0        2                    142
4                       0          6                   9        6                    286

NMF does worse here (ARI 0.333), and the crosstab shows why. Topic 1 is geb pitt cadre dsl chastity skepticism shameful intellect: the signature of one very active sci.med poster, "Skepticism is the chastity of the intellect, and it is shameful to surrender it too soon", with his e-mail address at cadre.dsl.pitt.edu. It repeats word for word at the end of dozens of his posts, so NMF spends a whole topic reconstructing it (79 medical posts end up there). The Middle East takes two topics (Armenia and Turkey; Israel), and cars, baseball and medicine are left sharing topic 0 (460 + 464 + 394 posts). With \(k\) fixed at 5, every topic spent on a signature or a sub-theme is one topic fewer for the rest. In the lesson's corpus NMF beat LDA; here it is the other way round: neither model wins everywhere.

Step 4 — BERTopic step 1: embed the posts

from sentence_transformers import SentenceTransformer

encoder = SentenceTransformer('all-MiniLM-L6-v2')      # downloaded once, about 90 MB
E = encoder.encode(docs, show_progress_bar=True)       # the slow step: about 1 minute on a CPU
print(E.shape)

(2613, 384)

One 384-dimensional vector per post. This is the only step that needs a neural network; the next ones take seconds. If you rerun the notebook, keep E and skip this cell.

Step 5 — BERTopic step 2: reduce with UMAP (and look)

import matplotlib.pyplot as plt
from umap import UMAP

umap_model = UMAP(n_neighbors=15, n_components=5, min_dist=0.0, metric='cosine', random_state=42)
Z = umap_model.fit_transform(E)                         # 5 dimensions, for the clustering
print(Z.shape)

Z2 = UMAP(n_neighbors=15, n_components=2, min_dist=0.0, metric='cosine',
          random_state=42).fit_transform(E)             # 2 dimensions, only to look
for name, color in zip(forums, ['#58a6ff', '#f0883e', '#3fb950', '#ff7b72', 'gray']):
    m = forum == name
    plt.scatter(Z2[m, 0], Z2[m, 1], s=4, color=color, label=name)
plt.legend(markerscale=4, fontsize=8)
plt.title('posts after UMAP, colored by the forum the models never saw')
plt.show()

(2613, 5)

The 5-dimensional Z is what HDBSCAN will cluster. The 2-dimensional version is only for the eye (as in the dimensionality reduction lesson, do not read distances between far-apart islands). You should see one island per forum, well apart from the others, with the Middle East in more than one piece.

Step 6 — BERTopic step 3: cluster with HDBSCAN

from hdbscan import HDBSCAN

hdbscan_model = HDBSCAN(min_cluster_size=20, metric='euclidean',   # BERTopic's default is 10
                        cluster_selection_method='eom', prediction_data=True)
labels = hdbscan_model.fit_predict(Z)
print(labels.max() + 1, 'clusters |', (labels == -1).sum(), 'outliers')
print(pd.crosstab(labels, forum, rownames=['cluster'], colnames=['forum']).to_string())

7 clusters | 59 outliers
forum    comp.windows.x  rec.autos  rec.sport.baseball  sci.med  talk.politics.mideast
cluster
-1                    3          2                   8       23                     23
 0                    0          3                 470        1                      0
 1                  531          3                   0        9                      1
 2                    1        481                   4        6                      0
 3                    0          0                   1        0                    152
 4                    1          4                   2      498                      4
 5                    3          7                   7        9                    326
 6                   10         10                   3        5                      2

7 clusters and 59 outliers (2% of the posts, against 20% in the lesson's corpus: these forums are better separated). min_cluster_size is 20 here, not BERTopic's default of 10: on this corpus, 10 sits on one of the cliffs of the pitfalls section, where differences in the sixth decimal of the embeddings change the number of topics (item 1 of Your turn lets you check). Four clusters are one forum each: 470 baseball posts in cluster 0, 531 X Window in cluster 1, 481 cars in cluster 2, 498 medicine in cluster 4. The Middle East splits into two clusters (326 posts in cluster 5, 152 in cluster 3), and cluster 6 is a small mix of everything (30 posts). The outliers come mostly from medicine and the Middle East, 23 each.

Step 7 — BERTopic step 4 and the one-call model

from bertopic import BERTopic

vectorizer = CountVectorizer(stop_words='english', token_pattern=r'(?u)\b[a-zA-Z]{3,}\b')
topic_model = BERTopic(embedding_model=encoder, umap_model=umap_model,
                       hdbscan_model=hdbscan_model, vectorizer_model=vectorizer)
topics = np.array(topic_model.fit_transform(docs, embeddings=E)[0])
print("same partition as step 6 (ARI):", round(adjusted_rand_score(labels, topics), 3))
print(topic_model.get_topic_info()[['Topic', 'Count', 'Name']].to_string(index=False))

same partition as step 6 (ARI): 1.0
 Topic  Count                                Name
    -1     59 -1_photography_kirlian_anania_music
     0    544           0_file_window_program_use
     1    509          1_medical_edu_patients_msg
     2    492              2_car_cars_engine_like
     3    474              3_year_game_team_games
     4    352          4_israel_israeli_jews_arab
     5    153 5_armenian_armenians_turkish_people
     6     30            6_com_health_tobacco_edu

The same partition as step 6, renumbered by size, now with names. The Middle East turns out to be two different conflicts: Israel and the Arab world (topic 4, 352 posts) and Armenia and Turkey (topic 5, 153). Topic 6 is the mixed one, 30 posts from all five forums, named by com, health, tobacco, edu: requests, mailing-list announcements and newsletters, grouped by their form (addresses, subscription instructions) more than by a subject. The outlier topic is named after a small medical side-thread on Kirlian photography, too few posts to make a cluster of 20. None of this was asked for: the number of topics came from min_cluster_size and the density of the embeddings.

Step 8 — inspect topics

found, sim = topic_model.find_topics('car engine', top_n=1)   # search topics by meaning
t = found[0]
print(t, [w for w, _ in topic_model.get_topic(t)][:6], round(float(sim[0]), 3))
for doc in topic_model.get_representative_docs(t)[:2]:
    print('  -', ' '.join(doc.split())[:90])

new = ["The doctor prescribed antibiotics for my sinus infection",
       "Which V8 engine should I put in my old pickup truck?"]
print([int(t) for t in topic_model.transform(new)[0]])

2 ['car', 'cars', 'engine', 'like', 'just', 'good'] 0.466
  - While I don't read normally read this group, I was looking for Valentine radar information
  - *nnnnnnnng* Thank you for playing, I cannot agree with this. I believed this and to put it
[1, 2]

find_topics embeds the query and returns the most similar topics: car engine matches topic 2 with a cosine of 0.466. get_representative_docs gives up to three typical posts of a topic (the first one is about a radar detector, something car owners discuss). transform embeds new documents and runs them through the fitted UMAP and HDBSCAN: the antibiotics sentence goes to topic 1 (medicine) and the V8 question to topic 2 (cars). This is how a fitted topic model labels next week's tickets without being refitted.

Step 9 — compare with the forums

from sklearn.metrics import homogeneity_score

def scores(name, lab):
    inside = lab >= 0
    return dict(model=name, topics=len(set(lab[inside])), outliers=int((~inside).sum()),
                ARI=round(adjusted_rand_score(forum, lab), 3),
                homogeneity=round(homogeneity_score(forum[inside], lab[inside]), 3))

rows = [scores('LDA', lda_topic), scores('NMF', nmf_topic), scores('BERTopic', topics)]
topic_model.reduce_topics(docs, nr_topics=6)            # merge down to 5 topics + the outlier topic
merged = np.array(topic_model.topics_)
rows.append(scores('BERTopic, merged', merged))
print(pd.DataFrame(rows).to_string(index=False))
print(pd.crosstab(merged, forum, rownames=['topic'], colnames=['forum']).to_string())

           model  topics  outliers   ARI  homogeneity
             LDA       5         0 0.603        0.611
             NMF       5         0 0.333        0.420
        BERTopic       7        59 0.848        0.902
BERTopic, merged       5        59 0.891        0.888
forum  comp.windows.x  rec.autos  rec.sport.baseball  sci.med  talk.politics.mideast
topic
-1                  3          2                   8       23                     23
 0                531          3                   0        9                      1
 1                 11        491                   7       11                      2
 2                  1          4                   2      498                      4
 3                  3          7                   8        9                    478
 4                  0          3                 470        1                      0

With its own 7 topics, BERTopic agrees with the forums far better than LDA and NMF (ARI 0.848 against 0.603 and 0.333), even though ARI penalizes it for splitting the Middle East in two. reduce_topics merges the most similar topics (by the cosine of their c-TF-IDF vectors) until 5 remain, plus −1. The merged topics map one to one onto the forums: 531 of the 544 posts of topic 0 are X Window, 491 cars in topic 1, 498 medicine in topic 2, 478 Middle East in topic 3, 470 baseball in topic 4; ARI rises to 0.891. The comparison is not entirely fair: BERTopic's scores leave out its 59 outliers, and LDA and NMF have to place every post. Step 10 removes that advantage.

Step 10 — the outliers

out = merged == -1
print(pd.Series(forum[out]).value_counts().to_string())
for doc in np.array(docs)[out][:3]:
    print('  -', ' '.join(doc.split())[:90])

new_topics = np.array(topic_model.reduce_outliers(docs, merged, strategy='embeddings', embeddings=E))
topic_model.update_topics(docs, topics=new_topics.tolist(), vectorizer_model=vectorizer)
print(scores('BERTopic, merged, no outliers', new_topics))

sci.med                  23
talk.politics.mideast    23
rec.sport.baseball        8
comp.windows.x            3
rec.autos                 2
  - In-reply-to: todamhyp@charles.unlv.edu's message of Mon, 19 Apr 93 20:56:15 GMT Newsgroups
  - Yes, I know this is not Rec.music, (as someone has already pointed out, thanks I know that
  - What _is_ your problem? Hite's post wasn't a flame. It was a correction of *your* error. Y
{'model': 'BERTopic, merged, no outliers', 'topics': 5, 'outliers': 0, 'ARI': 0.899, 'homogeneity': 0.868}

The outliers come mostly from medicine and the Middle East (23 each), and reading a few shows why: a cross-posted thread, replies arguing about other replies, posts with little topic of their own. reduce_outliers with strategy='embeddings' gives each one the topic whose centroid embedding is closest; update_topics then recomputes the topic words with the new members (BERTopic warns that further topic reduction would be unsafe after this; it is the last step). On all 2,613 posts, ARI is 0.899 and homogeneity 0.868, still far ahead of LDA (0.603) and NMF (0.333).

Your turn

1. Reproducibility. Refit steps 5 to 7 with random_state 0, 1 and 2 in the 5-dimensional UMAP, keeping E. For each seed, record the number of topics, the number of outliers and the ARI with the forums; then compute the ARI between the three runs. Rerun one seed twice and check that the result is identical. Then set min_cluster_size back to 10 and repeat, also adding a noise of 1e-6 to E as in the pitfalls section. Which setting moves more? What would you report to someone who asks "how many topics are there"?
2. Granularity. Refit with min_cluster_size 5, 20 and 50 (same Z). Compare with keeping 10 and calling reduce_topics(docs, nr_topics=...). Which route gives more readable topics at about 8 topics?
3. Representation. Refit with a CountVectorizer() without stop words, then call update_topics with representation_model=KeyBERTInspired() (from bertopic.representation). Which topics change their words the most? Check that the assignments do not change.
4. Short texts. Cut every post to its first 15 words and repeat steps 2, 3 and 7 (embedding again). Which model survives best? Then try a Portuguese corpus of your own (news headlines, reviews), with SentenceTransformer('paraphrase-multilingual-MiniLM-L12-v2') and a Portuguese stop-word list.

Class materials

References

• Blei, D. M.; Ng, A. Y.; Jordan, M. I. "Latent Dirichlet Allocation." JMLR 3 (2003): 993–1022 — the generative story and its variational inference. full text
• Lee, D. D.; Seung, H. S. "Learning the Parts of Objects by Non-negative Matrix Factorization." Nature 401 (1999): 788–791 — NMF and parts-based representations. DOI
• Grootendorst, M. "BERTopic: Neural Topic Modeling with a Class-based TF-IDF Procedure." arXiv (2022). arXiv:2203.05794
• Reimers, N.; Gurevych, I. "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks." EMNLP-IJCNLP (2019) — the sentence-transformers behind step 1. arXiv:1908.10084
• McInnes, L.; Healy, J.; Melville, J. "UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction." arXiv (2018). arXiv:1802.03426
• Campello, R. J. G. B.; Moulavi, D.; Sander, J. "Density-Based Clustering Based on Hierarchical Density Estimates." PAKDD (2013), LNCS 7819: 160–172 — HDBSCAN. DOI
• Röder, M.; Both, A.; Hinneburg, A. "Exploring the Space of Topic Coherence Measures." WSDM (2015): 399–408 — coherence measures compared with human judgments. DOI
• Chang, J.; Boyd-Graber, J.; Gerrish, S.; Wang, C.; Blei, D. M. "Reading Tea Leaves: How Humans Interpret Topic Models." NeurIPS 22 (2009) — better perplexity does not mean more interpretable topics.
• Dieng, A. B.; Ruiz, F. J. R.; Blei, D. M. "Topic Modeling in Embedding Spaces." TACL 8 (2020): 439–453 — where the topic diversity score comes from. DOI

Quiz