Topic Modeling & BERTopic

Topic modeling answers a question that sounds simple: given thousands of documents, what are they about? Nobody labels the documents and nobody lists the themes in advance, so it is unsupervised learning, the text-world sibling of clustering. Typical uses: reading a year of customer reviews or support tickets without reading them, organizing a news archive, following what people discuss on social media, mapping a scientific literature.

This lesson builds three topic models, from the classical to the modern. LDA tells a probabilistic story about how words are written. NMF factorizes the TF-IDF matrix of the text representation lessons. BERTopic is almost entirely made of pieces you already know: sentence embeddings, UMAP, HDBSCAN and a variant of TF-IDF. Two corpora run through the whole lesson: eleven sentences written by hand, small enough to follow with a calculator, and 2,155 real posts from four internet forums, whose forum we know but hide from the models, so that we can check what they find. Every number below comes from the code shown next to it, and each figure is drawn by a short script that sits next to its page.

How this lesson is organized

The lesson has five pages, this overview and E3.1 to E3.4, and every page has two approaches, each with its own slides and text. Read the fundamental approach of the five pages first, in order, and then do the fundamental lab at the end of the fundamental approach of this page; the advanced approach of each page builds on its fundamental one, and the advanced lab closes the lesson. The quiz at the end covers the fundamental approach.

Both labs use the same lab notebook: the fundamental part runs from step 1 to 7, the advanced part from step 8 to 10. It needs bertopic, which a Colab cell installs in a minute.

The pages of this lesson

The lesson continues in four pages, meant to be read in order. Each page's code continues the previous one, and each page also opens with a setup block that rebuilds what it needs, so it can be run on its own.

Fundamental approach

One question drives this part: given thousands of documents that nobody has labeled, what are they about? There are two answers to "what is a topic?", and the three models of the lesson take one or the other. This part builds each model, first on eleven hand-written sentences and then on 2,155 forum posts, and measures what each one finds against the forums the models never saw.

This part follows one path in five stages, drawn below: 1. what is a topic?, the two answers and a hand-made corpus; 2. LDA, a topic as a distribution over words, told as a probabilistic story; 3. NMF, the same answer with linear algebra; 4. BERTopic, a topic as a cluster of documents; 5. compare and choose, the three models measured side by side. Stage 1 is on this page and each of the other stages is the fundamental approach of its own page. Each stage links to its section or page, and in the slides a strip at the bottom shows which stage you are in.

Fundamental approach slides

Click the slides and use the ← → arrows to navigate; F opens full screen and Esc shows every slide. Open the slides in a new tab

1. What is a topic?

Stage 1. There are two answers, and the models of this lesson use one or the other:

• a topic is a distribution over words. "Sports" gives high probability to team, game, goal and almost none to oven. A document is a mixture of topics: 80% sports and 20% space, say. LDA works this way, and NMF too, with non-negative weights instead of probabilities;
• a topic is a cluster of documents. Each document belongs to one topic (or to none), and the topic's words are computed afterwards, to describe the cluster. BERTopic works this way.

Both start from documents turned into numbers. Here is the hand-made corpus, as the bag-of-words counts of the text representation lessons:

import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer

docs_toy = [
    "the team won the game with a late goal",                 # sports
    "fans cheered when the team scored a goal",
    "the coach said the game was a fair win for the team",
    "the rocket launch put a satellite in orbit",             # space
    "nasa delayed the rocket launch to the moon",
    "the satellite reached orbit around the moon",
    "bake the bread in a hot oven",                           # cooking
    "knead the dough and bake the bread slowly",
    "fresh bread from the oven smells great",
    "the team watched the rocket launch before the game",     # sports + space
    "astronauts ate fresh bread on the moon",                 # space + cooking
]
cv_toy = CountVectorizer(stop_words='english')
X_toy = cv_toy.fit_transform(docs_toy)
words_toy = cv_toy.get_feature_names_out()
print(X_toy.shape, '| word tokens kept:', X_toy.sum())
counts = pd.DataFrame(X_toy.toarray(), columns=words_toy)
print(counts[['team', 'game', 'goal', 'rocket', 'launch', 'moon', 'bread', 'bake']])

(11, 33) | word tokens kept: 53
    team  game  goal  rocket  launch  moon  bread  bake
0      1     1     1       0       0     0      0     0
1      1     0     1       0       0     0      0     0
2      1     1     0       0       0     0      0     0
3      0     0     0       1       1     0      0     0
4      0     0     0       1       1     1      0     0
5      0     0     0       0       0     1      0     0
6      0     0     0       0       0     0      1     1
7      0     0     0       0       0     0      1     1
8      0     0     0       0       0     0      1     0
9      1     1     0       1       1     0      0     0
10     0     0     0       0       0     1      1     0

After the English stop words (the, a, with, ...) are removed, 11 sentences keep 53 words drawn from a vocabulary of 33. Three themes are easy to see, and the last two sentences mix them. A topic model receives only this matrix, without the comments, and must find the themes.

The path continues on the fundamental approach of E3.1 LDA, then E3.2, E3.3 and E3.4, in order. Come back here for the summary and the lab.

2. Summary: topic modeling in five lines

Read this after the fundamental approach of the four pages.

1. A topic is a distribution over words, with every document a mixture of topics (LDA, NMF), or a cluster of documents named afterwards by its words (BERTopic).
2. LDA tells a generative story: every word slot draws a topic from the document's mixture \(\theta_d\), then a word from that topic, so \(p(w \mid d) = \sum_k \theta_{d,k}\, \varphi_{k,w}\). Its Dirichlet prior \(\alpha\) says how mixed a typical document is.
3. NMF writes \(X \approx WH\) with nothing negative: each topic is a whole theme, each document a sum of parts, and the weights can be read directly.
4. BERTopic chains four steps: embed, reduce with UMAP, cluster with HDBSCAN, name each cluster with c-TF-IDF. The number of topics comes out of the data, and a document may get no topic (−1).
5. No model wins everywhere. Measure the topics against something the models never saw, and read them.

Fundamental lab — topics in five forums, three ways

The problem. A moderator inherits an archive of 2,613 posts from five forums of the 20 Newsgroups collection, mixed together, and wants to know what people talk about without reading them. The forum of each post is known, so the moderator can check the answer, but the models never see it. This part fits the three models of the lesson: LDA and NMF in steps 2 and 3, then BERTopic one step at a time and in one call, in steps 4 to 7. Using, comparing and cleaning the topics are left to the advanced lab.

Step 1 — load five forums

Predict. Five forums of about 500 posts each: windowing software, cars, baseball, medicine and Middle East politics. Which of them might really hold more than one subject? Which two might a word-count model confuse?

import numpy as np
import pandas as pd
from sklearn.datasets import fetch_20newsgroups

forums = ['comp.windows.x', 'rec.autos', 'rec.sport.baseball', 'sci.med', 'talk.politics.mideast']
ng = fetch_20newsgroups(subset='train', categories=forums, random_state=0,
                        remove=('headers', 'footers', 'quotes'))
keep = np.array([len(text.split()) >= 20 for text in ng.data])
docs = [text for text, k in zip(ng.data, keep) if k]
forum = np.array(ng.target_names)[ng.target[keep]]     # hidden from the models; used to check them
print(len(docs), 'posts')
print(pd.Series(forum).value_counts())

2613 posts
sci.med                  551
comp.windows.x           549
rec.autos                510
talk.politics.mideast    508
rec.sport.baseball       495
Name: count, dtype: int64

2,613 posts, between 495 and 551 per forum. The forum names are kept in forum to check the results; none of the models sees them. talk.politics.mideast discusses several different conflicts, which will matter from step 6 on.

Step 2 — LDA

Predict. Five topics for five forums. Will every LDA topic be one forum? Which forum is most likely to hide behind generic words?

from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics import adjusted_rand_score

settings = dict(stop_words='english', min_df=5, max_df=0.5, token_pattern=r'(?u)\b[a-zA-Z]{3,}\b')
cv = CountVectorizer(**settings)
X = cv.fit_transform(docs)
vocab = cv.get_feature_names_out()

def top_words(components, n=8):
    return [' '.join(vocab[row.argsort()[::-1][:n]]) for row in components]

lda = LatentDirichletAllocation(n_components=5, random_state=0).fit(X)
lda_topic = lda.transform(X).argmax(axis=1)
print(*top_words(lda.components_), sep='\n')
print(f"ARI with the forums: {adjusted_rand_score(forum, lda_topic):.3f}")

car entry like don file just output use
year good think don game just team like
people armenian said armenians israel turkish jews know
window edu com server use available motif file
edu health medical use university research patients disease
ARI with the forums: 0.603

Five topics for five forums. The X Window System (window, server, motif), the Middle East and medicine are recognizable; baseball hides behind generic words (year good think don game), and the first topic mixes cars with software words (entry, output, file). Sending each post to its largest topic gives an ARI of 0.603 with the forums.

Step 3 — NMF

Predict. On the lesson's four forums NMF beat LDA by far (0.767 against 0.456). Will it win again here, with the same settings?

from sklearn.decomposition import NMF
from sklearn.feature_extraction.text import TfidfVectorizer

T = TfidfVectorizer(**settings).fit_transform(docs)     # same vocabulary as X
nmf = NMF(n_components=5, init='nndsvda', max_iter=500, random_state=0).fit(T)
nmf_topic = nmf.transform(T).argmax(axis=1)
print(*top_words(nmf.components_), sep='\n')
print(f"ARI with the forums: {adjusted_rand_score(forum, nmf_topic):.3f}")
print(pd.crosstab(nmf_topic, forum, rownames=['NMF topic'], colnames=['forum']).to_string())

car don just year good think like know
geb pitt cadre dsl chastity skepticism shameful intellect
window server motif use thanks application widget program
armenian armenians turkish people armenia turks turkey genocide
israel israeli jews arab arabs jewish peace lebanese
ARI with the forums: 0.333
forum      comp.windows.x  rec.autos  rec.sport.baseball  sci.med  talk.politics.mideast
NMF topic
0                      12        460                 464      394                     66
1                       0          2                   0       79                      0
2                     537         41                  22       70                     14
3                       0          1                   0        2                    142
4                       0          6                   9        6                    286

NMF does worse here (ARI 0.333), and the crosstab shows why. Topic 1 is geb pitt cadre dsl chastity skepticism shameful intellect: the signature of one very active sci.med poster, "Skepticism is the chastity of the intellect, and it is shameful to surrender it too soon", with his e-mail address at cadre.dsl.pitt.edu. It repeats word for word at the end of dozens of his posts, so NMF spends a whole topic reconstructing it (79 medical posts end up there). The Middle East takes two topics (Armenia and Turkey; Israel), and cars, baseball and medicine are left sharing topic 0 (460 + 464 + 394 posts). With \(k\) fixed at 5, every topic spent on a signature or a sub-theme is one topic fewer for the rest. In the lesson's corpus NMF beat LDA; here it is the other way round: neither model wins everywhere.

Simulation: more topics for NMF. The reading above blamed NMF's low ARI on a budget of five topics, one of them spent on a signature. Give NMF a bigger budget: refit it with \(k = 6\) and \(k = 8\) and print the top words and the ARI of each. Does the signature topic go away? Do cars, baseball and medicine get topics of their own?

for k in [6, 8]:
    nmf_k = NMF(n_components=k, init='nndsvda', max_iter=500, random_state=0).fit(T)
    label_k = nmf_k.transform(T).argmax(axis=1)
    print(f"k = {k}: ARI with the forums {adjusted_rand_score(forum, label_k):.3f}")
    print(*top_words(nmf_k.components_, 6), sep='\n')

k = 6: ARI with the forums 0.474
year team game games runs pitching
geb pitt cadre dsl chastity skepticism
window server motif use application widget
armenian armenians turkish people armenia turks
israel israeli jews arab arabs jewish
car just don like know good
k = 8: ARI with the forums 0.511
year team game games runs pitching
geb dsl cadre pitt chastity skepticism
server motif thanks widget use edu
armenian armenians turkish armenia people turks
israel israeli jews arab arabs jewish
car cars engine dealer new like
msg don people know just food
window manager application windows display expose

Step 4 — BERTopic step 1: embed the posts

Predict. A post of 300 words and a post of 30 words: will their embeddings have the same number of numbers? What does that say about how a long post is summarized?

from sentence_transformers import SentenceTransformer

encoder = SentenceTransformer('all-MiniLM-L6-v2')      # downloaded once, about 90 MB
E = encoder.encode(docs, show_progress_bar=True)       # the slow step: about 1 minute on a CPU
print(E.shape)

(2613, 384)

One 384-dimensional vector per post, whatever its length. This is the only step that needs a neural network; the next ones take seconds. If you rerun the notebook, keep E and skip this cell.

Step 5 — BERTopic step 2: reduce with UMAP (and look)

Predict. The code reduces the embeddings twice: to 5 dimensions and to 2. Why not use the 2-dimensional version for everything? In the picture, which forum do you expect in more than one piece?

import matplotlib.pyplot as plt
from umap import UMAP

umap_model = UMAP(n_neighbors=15, n_components=5, min_dist=0.0, metric='cosine', random_state=42)
Z = umap_model.fit_transform(E)                         # 5 dimensions, for the clustering
print(Z.shape)

Z2 = UMAP(n_neighbors=15, n_components=2, min_dist=0.0, metric='cosine',
          random_state=42).fit_transform(E)             # 2 dimensions, only to look
for name, color in zip(forums, ['#58a6ff', '#f0883e', '#3fb950', '#ff7b72', 'gray']):
    m = forum == name
    plt.scatter(Z2[m, 0], Z2[m, 1], s=4, color=color, label=name)
plt.legend(markerscale=4, fontsize=8)
plt.title('posts after UMAP, colored by the forum the models never saw')
plt.show()

(2613, 5)

The 5-dimensional Z is what HDBSCAN will cluster. The 2-dimensional version is only for the eye (as in the dimensionality reduction lesson, do not read distances between far-apart islands). You should see one island per forum, well apart from the others, with the Middle East in more than one piece.

Step 6 — BERTopic step 3: cluster with HDBSCAN

Predict. Nobody tells HDBSCAN how many clusters to find. Will it find exactly five? Which forums will give it the most outliers (−1)?

from hdbscan import HDBSCAN

hdbscan_model = HDBSCAN(min_cluster_size=20, metric='euclidean',   # BERTopic's default is 10
                        cluster_selection_method='eom', prediction_data=True)
labels = hdbscan_model.fit_predict(Z)
print(labels.max() + 1, 'clusters |', (labels == -1).sum(), 'outliers')
print(pd.crosstab(labels, forum, rownames=['cluster'], colnames=['forum']).to_string())

7 clusters | 59 outliers
forum    comp.windows.x  rec.autos  rec.sport.baseball  sci.med  talk.politics.mideast
cluster
-1                    3          2                   8       23                     23
 0                    0          3                 470        1                      0
 1                  531          3                   0        9                      1
 2                    1        481                   4        6                      0
 3                    0          0                   1        0                    152
 4                    1          4                   2      498                      4
 5                    3          7                   7        9                    326
 6                   10         10                   3        5                      2

7 clusters and 59 outliers (2% of the posts, against 20% in the lesson's corpus: these forums are better separated). min_cluster_size is 20 here, not BERTopic's default of 10: on this corpus, 10 sits on one of the cliffs of the pitfalls section, where differences in the sixth decimal of the embeddings change the number of topics (item 1 of Your turn (advanced) lets you check). Four clusters are one forum each: 470 baseball posts in cluster 0, 531 X Window in cluster 1, 481 cars in cluster 2, 498 medicine in cluster 4. The Middle East splits into two clusters (326 posts in cluster 5, 152 in cluster 3), and cluster 6 is a small mix of everything (30 posts). The outliers come mostly from medicine and the Middle East, 23 each.

Simulation: the embeddings or HDBSCAN? This crosstab is much cleaner than the ones of steps 2 and 3. Who deserves the credit, the embeddings or HDBSCAN? Run plain k-means with 5 clusters on three representations of the same posts, the TF-IDF matrix T, the embeddings E and the 5-dimensional Z, and compare their ARI with the forums.

from sklearn.cluster import KMeans

for name, V in [('TF-IDF T', T), ('embeddings E', E), ('UMAP Z', Z)]:
    km = KMeans(n_clusters=5, n_init=10, random_state=0).fit_predict(V)
    print(f"k-means with 5 clusters on {name:12s}  ARI {adjusted_rand_score(forum, km):.3f}")

k-means with 5 clusters on TF-IDF T      ARI 0.374
k-means with 5 clusters on embeddings E  ARI 0.863
k-means with 5 clusters on UMAP Z        ARI 0.892

Step 7 — BERTopic step 4 and the one-call model

Predict. BERTopic receives the same UMAP and HDBSCAN objects and the embeddings of step 4. Will its partition be the one of step 6? What will give the topics their names?

from bertopic import BERTopic

vectorizer = CountVectorizer(stop_words='english', token_pattern=r'(?u)\b[a-zA-Z]{3,}\b')
topic_model = BERTopic(embedding_model=encoder, umap_model=umap_model,
                       hdbscan_model=hdbscan_model, vectorizer_model=vectorizer)
topics = np.array(topic_model.fit_transform(docs, embeddings=E)[0])
print("same partition as step 6 (ARI):", round(adjusted_rand_score(labels, topics), 3))
print(topic_model.get_topic_info()[['Topic', 'Count', 'Name']].to_string(index=False))

same partition as step 6 (ARI): 1.0
 Topic  Count                                Name
    -1     59 -1_photography_kirlian_anania_music
     0    544           0_file_window_program_use
     1    509          1_medical_edu_patients_msg
     2    492              2_car_cars_engine_like
     3    474              3_year_game_team_games
     4    352          4_israel_israeli_jews_arab
     5    153 5_armenian_armenians_turkish_people
     6     30            6_com_health_tobacco_edu

The same partition as step 6, renumbered by size, now with names. The Middle East turns out to be two different conflicts: Israel and the Arab world (topic 4, 352 posts) and Armenia and Turkey (topic 5, 153). Topic 6 is the mixed one, 30 posts from all five forums, named by com, health, tobacco, edu: requests, mailing-list announcements and newsletters, grouped by their form (addresses, subscription instructions) more than by a subject. The outlier topic is named after a small medical side-thread on Kirlian photography, too few posts to make a cluster of 20. None of this was asked for: the number of topics came from min_cluster_size and the density of the embeddings.

To close the fundamental part. In one sentence each: (1) why did NMF spend a whole topic on one person's signature? (2) Which part of BERTopic deserves most of the credit for its clean clusters, according to the second simulation? (3) What are BERTopic's four steps, and which lesson does each one come from?

Your turn (fundamental)

At home:

1. LDA's prior. Refit step 2 with doc_topic_prior=0.05 and with doc_topic_prior=1.0. Do the top words change? And the ARI with the forums?
2. Remove the signature. Add the eight words of NMF's topic 1 (geb, pitt, cadre, dsl, chastity, skepticism, shameful, intellect) to the stop words, with stop_words=list(ENGLISH_STOP_WORDS) + [...] (ENGLISH_STOP_WORDS comes from sklearn.feature_extraction.text), and refit step 3 with 5 topics. Does medicine get a topic of its own? What happens to the ARI?
3. Color by topic. Redraw the 2-dimensional map of step 5 colored by the topics of step 7 instead of the forums. Which islands did BERTopic split, and where are the outliers?

Advanced approach

The fundamental approach built three topic models and read their topics. This part asks whether those topics can be trusted. On each page it goes one level deeper: does fitting LDA recover topics that are known, and what happens when the words are too few? How many topics should there be, and is any score a good judge? Can BERTopic's names be reproduced by hand and improved without re-clustering? Which pitfalls catch most topic-modeling projects? The advanced lab closes the lesson by using the fitted model as a tool and comparing the three models fairly.

This part follows one path in five stages, drawn below: 1. under LDA's hood; 2. how many topics; 3. name and rename; 4. the pitfalls; 5. use and check. Stages 1 to 4 are the advanced approaches of the four pages, in order, and stage 5 is the advanced lab on this page. Each stage links to its page or section, and in the slides a strip at the bottom shows which stage you are in.

Advanced approach slides

Click the slides and use the ← → arrows to navigate; F opens full screen and Esc shows every slide. Open the slides in a new tab

Advanced lab — use, compare and clean the topics

Stage 5. The fundamental lab ended with a BERTopic model of 7 topics, an LDA and an NMF of 5. This part uses the fitted model as a tool (search, new posts), compares the three models with the forums, first as they are and then on equal terms, and decides what to do with the outliers. Continue in the same notebook, in the same session: the steps below use docs, forum, E, lda_topic, nmf_topic, topics, topic_model and vectorizer from steps 1 to 7.

Step 8 — inspect topics

Predict. find_topics('car engine') searches the topics by meaning. Will it find the cars topic even if the word engine were not among its top words? And will two new sentences, never seen in training, land in sensible topics?

found, sim = topic_model.find_topics('car engine', top_n=1)   # search topics by meaning
t = found[0]
print(t, [w for w, _ in topic_model.get_topic(t)][:6], round(float(sim[0]), 3))
for doc in topic_model.get_representative_docs(t)[:2]:
    print('  -', ' '.join(doc.split())[:90])

new = ["The doctor prescribed antibiotics for my sinus infection",
       "Which V8 engine should I put in my old pickup truck?"]
print([int(t) for t in topic_model.transform(new)[0]])

2 ['car', 'cars', 'engine', 'like', 'just', 'good'] 0.466
  - While I don't read normally read this group, I was looking for Valentine radar information
  - *nnnnnnnng* Thank you for playing, I cannot agree with this. I believed this and to put it
[1, 2]

find_topics embeds the query and returns the most similar topics: car engine matches topic 2 with a cosine of 0.466. get_representative_docs gives up to three typical posts of a topic (the first one is about a radar detector, something car owners discuss). transform embeds new documents and runs them through the fitted UMAP and HDBSCAN: the antibiotics sentence goes to topic 1 (medicine) and the V8 question to topic 2 (cars). This is how a fitted topic model labels next week's tickets without being refitted.

Simulation: next week's posts. transform labeled two sentences. Label a whole new batch: the test part of the same five forums (fetch_20newsgroups(subset='test', ...), with the same filters as step 1), which the model never saw. How many posts land in topic −1, and how well do the topics agree with the forums?

test = fetch_20newsgroups(subset='test', categories=forums, random_state=0,
                          remove=('headers', 'footers', 'quotes'))
keep_test = np.array([len(text.split()) >= 20 for text in test.data])
docs_test = [text for text, k in zip(test.data, keep_test) if k]
forum_test = np.array(test.target_names)[test.target[keep_test]]
topics_test = np.array(topic_model.transform(docs_test)[0])    # embed, reduce, cluster: no refit
print(len(docs_test), 'new posts |', (topics_test == -1).sum(), 'outliers')
print(f"ARI with their forums: {adjusted_rand_score(forum_test, topics_test):.3f}")
print(pd.crosstab(topics_test, forum_test, rownames=['topic'], colnames=['forum']).to_string())

1729 new posts | 95 outliers
ARI with their forums: 0.829
forum  comp.windows.x  rec.autos  rec.sport.baseball  sci.med  talk.politics.mideast
topic
-1                  3          3                   6       21                     62
 0                347          0                   1        4                      2
 1                  0          0                   4      311                      3
 2                  4        328                   1        5                      1
 3                  0          0                 320        3                      3
 4                  0          0                   6        4                    189
 5                  0          1                   0        2                     84
 6                  2          3                   1        5                      0

Step 9 — compare with the forums

Predict. BERTopic has 7 topics and 59 outliers; LDA and NMF have 5 topics and place every post. Which model will agree best with the forums? And is it a fair contest?

from sklearn.metrics import homogeneity_score

def scores(name, lab):
    inside = lab >= 0
    return dict(model=name, topics=len(set(lab[inside])), outliers=int((~inside).sum()),
                ARI=round(adjusted_rand_score(forum, lab), 3),
                homogeneity=round(homogeneity_score(forum[inside], lab[inside]), 3))

rows = [scores('LDA', lda_topic), scores('NMF', nmf_topic), scores('BERTopic', topics)]
topic_model.reduce_topics(docs, nr_topics=6)            # merge down to 5 topics + the outlier topic
merged = np.array(topic_model.topics_)
rows.append(scores('BERTopic, merged', merged))
print(pd.DataFrame(rows).to_string(index=False))
print(pd.crosstab(merged, forum, rownames=['topic'], colnames=['forum']).to_string())

           model  topics  outliers   ARI  homogeneity
             LDA       5         0 0.603        0.611
             NMF       5         0 0.333        0.420
        BERTopic       7        59 0.848        0.902
BERTopic, merged       5        59 0.891        0.888
forum  comp.windows.x  rec.autos  rec.sport.baseball  sci.med  talk.politics.mideast
topic
-1                  3          2                   8       23                     23
 0                531          3                   0        9                      1
 1                 11        491                   7       11                      2
 2                  1          4                   2      498                      4
 3                  3          7                   8        9                    478
 4                  0          3                 470        1                      0

With its own 7 topics, BERTopic agrees with the forums far better than LDA and NMF (ARI 0.848 against 0.603 and 0.333), even though ARI penalizes it for splitting the Middle East in two. reduce_topics merges the most similar topics (by the cosine of their c-TF-IDF vectors) until 5 remain, plus −1. The merged topics map one to one onto the forums: 531 of the 544 posts of topic 0 are X Window, 491 cars in topic 1, 498 medicine in topic 2, 478 Middle East in topic 3, 470 baseball in topic 4; ARI rises to 0.891. The comparison is not entirely fair: BERTopic's scores leave out its 59 outliers, and LDA and NMF have to place every post. Step 10 removes that advantage.

Simulation: a fair comparison. Remove BERTopic's advantage the other way round: score LDA and NMF only on the 2,554 posts that BERTopic placed in a topic, and BERTopic on the same posts. Do LDA and NMF come close to BERTopic once the hard posts are gone?

kept = merged >= 0                                     # the posts BERTopic placed in a topic
for name, lab in [('LDA', lda_topic), ('NMF', nmf_topic), ('BERTopic, merged', merged)]:
    print(f"{name:17s} ARI on the {kept.sum()} posts BERTopic kept: "
          f"{adjusted_rand_score(forum[kept], lab[kept]):.3f}")

LDA               ARI on the 2554 posts BERTopic kept: 0.616
NMF               ARI on the 2554 posts BERTopic kept: 0.346
BERTopic, merged  ARI on the 2554 posts BERTopic kept: 0.917

Step 10 — the outliers

Predict. Giving each outlier the topic of the nearest centroid removes topic −1. Will the agreement with the forums go up or down? And the purity of the topics?

out = merged == -1
print(pd.Series(forum[out]).value_counts().to_string())
for doc in np.array(docs)[out][:3]:
    print('  -', ' '.join(doc.split())[:90])

new_topics = np.array(topic_model.reduce_outliers(docs, merged, strategy='embeddings', embeddings=E))
topic_model.update_topics(docs, topics=new_topics.tolist(), vectorizer_model=vectorizer)
print(scores('BERTopic, merged, no outliers', new_topics))

sci.med                  23
talk.politics.mideast    23
rec.sport.baseball        8
comp.windows.x            3
rec.autos                 2
  - In-reply-to: todamhyp@charles.unlv.edu's message of Mon, 19 Apr 93 20:56:15 GMT Newsgroups
  - Yes, I know this is not Rec.music, (as someone has already pointed out, thanks I know that
  - What _is_ your problem? Hite's post wasn't a flame. It was a correction of *your* error. Y
{'model': 'BERTopic, merged, no outliers', 'topics': 5, 'outliers': 0, 'ARI': 0.899, 'homogeneity': 0.868}

The outliers come mostly from medicine and the Middle East (23 each), and reading a few shows why: a cross-posted thread, replies arguing about other replies, posts with little topic of their own. reduce_outliers with strategy='embeddings' gives each one the topic whose centroid embedding is closest; update_topics then recomputes the topic words with the new members (BERTopic warns that further topic reduction would be unsafe after this; it is the last step). On all 2,613 posts, ARI is 0.899 and homogeneity 0.868, still far ahead of LDA (0.603) and NMF (0.333).

To close the lesson. In one sentence each: (1) why is BERTopic's lead over LDA and NMF not an artifact of the posts it leaves out? (2) What does transform let you do with next week's posts, and what does it not need? (3) Why does homogeneity fall when the outliers are forced into topics, while the ARI rises?

Your turn (advanced)

At home, with more time:

1. Reproducibility. Refit steps 5 to 7 with random_state 0, 1 and 2 in the 5-dimensional UMAP, keeping E. For each seed, record the number of topics, the number of outliers and the ARI with the forums; then compute the ARI between the three runs. Rerun one seed twice and check that the result is identical. Then set min_cluster_size back to 10 and repeat, also adding a noise of 1e-6 to E as in the pitfalls section. Which setting moves more? What would you report to someone who asks "how many topics are there"?
2. Granularity. Refit with min_cluster_size 5, 20 and 50 (same Z). Compare with keeping 10 and calling reduce_topics(docs, nr_topics=...). Which route gives more readable topics at about 8 topics?
3. Representation. Refit with a CountVectorizer() without stop words, then call update_topics with representation_model=KeyBERTInspired() (from bertopic.representation). Which topics change their words the most? Check that the assignments do not change.
4. Short texts. Cut every post to its first 15 words and repeat steps 2, 3 and 7 (embedding again). Which model survives best? Then try a Portuguese corpus of your own (news headlines, reviews), with SentenceTransformer('paraphrase-multilingual-MiniLM-L12-v2') and a Portuguese stop-word list.

Class materials

References

• Blei, D. M.; Ng, A. Y.; Jordan, M. I. "Latent Dirichlet Allocation." JMLR 3 (2003): 993–1022 — the generative story and its variational inference. full text
• Lee, D. D.; Seung, H. S. "Learning the Parts of Objects by Non-negative Matrix Factorization." Nature 401 (1999): 788–791 — NMF and parts-based representations. DOI
• Grootendorst, M. "BERTopic: Neural Topic Modeling with a Class-based TF-IDF Procedure." arXiv (2022). arXiv:2203.05794
• Reimers, N.; Gurevych, I. "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks." EMNLP-IJCNLP (2019) — the sentence-transformers behind step 1. arXiv:1908.10084
• McInnes, L.; Healy, J.; Melville, J. "UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction." arXiv (2018). arXiv:1802.03426
• Campello, R. J. G. B.; Moulavi, D.; Sander, J. "Density-Based Clustering Based on Hierarchical Density Estimates." PAKDD (2013), LNCS 7819: 160–172 — HDBSCAN. DOI
• Röder, M.; Both, A.; Hinneburg, A. "Exploring the Space of Topic Coherence Measures." WSDM (2015): 399–408 — coherence measures compared with human judgments. DOI
• Chang, J.; Boyd-Graber, J.; Gerrish, S.; Wang, C.; Blei, D. M. "Reading Tea Leaves: How Humans Interpret Topic Models." NeurIPS 22 (2009) — better perplexity does not mean more interpretable topics.
• Dieng, A. B.; Ruiz, F. J. R.; Blei, D. M. "Topic Modeling in Embedding Spaces." TACL 8 (2020): 439–453 — where the topic diversity score comes from. DOI

Quiz

The quiz covers the fundamental approach of the five pages.