Topic Modeling & BERTopic

Topic modeling answers a deceptively simple question: given thousands of documents, what are they about? It is unsupervised — no labels, no predefined categories — making it the text-world sibling of clustering. Applications: mining customer reviews and support tickets, organizing news archives, monitoring social media, exploring scientific literature.

This lesson introduces the classical approach (LDA) and then BERTopic — a modern pipeline assembled almost entirely from techniques you have already learned in this course.

Classical topic modeling: LDA

Latent Dirichlet Allocation (Blei, Ng & Jordan, 2003) is a probabilistic generative model built on bag-of-words counts. It assumes each document was "written" by a random process:

1. each topic is a probability distribution over the vocabulary (topic "sports": high probability for game, team, score...);
2. each document is a mixture of topics (70% sports, 30% finance);
3. every word in a document is generated by first sampling a topic from the document's mixture, then sampling a word from that topic.

Fitting LDA inverts the process: given only the observed words, infer the topic–word distributions and the document–topic mixtures.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

X = CountVectorizer(max_df=0.9, min_df=5, stop_words='english').fit_transform(docs)
lda = LatentDirichletAllocation(n_components=10, random_state=0).fit(X)

LDA served the field for two decades, but it inherits every limitation of bag-of-words:

• word order and context are ignored; synonyms are unrelated dimensions;
• the number of topics \(k\) must be fixed in advance;
• short texts (tweets, ticket titles) give very sparse counts — LDA struggles;
• topics are often hard to interpret without heavy preprocessing (stop-word lists, stemming, tuning).

BERTopic: topic modeling on embeddings

BERTopic (Grootendorst, 2022) replaces the generative story with a geometric one: embed documents so that semantic similarity is spatial proximity, then find the dense regions. The pipeline is a composition of the last three lessons:

flowchart LR
    A[Documents] --> B["1. Sentence embeddings<br><small>SBERT — Text Representation</small>"]
    B --> C["2. Reduce dimensions<br><small>UMAP — Dimensionality Reduction</small>"]
    C --> D["3. Cluster<br><small>HDBSCAN — Clustering</small>"]
    D --> E["4. Describe topics<br><small>c-TF-IDF — this lesson</small>"]

Step 1 — Embed. Each document becomes a dense vector via a sentence-transformer. Documents about refunds and money back land close together even with disjoint vocabulary — the decisive advantage over LDA.

Step 2 — Reduce. Embeddings have 384–768 dimensions; density-based clustering suffers there (the curse of dimensionality). UMAP compresses to ~5 dimensions while preserving neighborhood structure.

Step 3 — Cluster. HDBSCAN finds clusters of varying shape and density — and crucially, it does not force every document into a topic: documents that fit nowhere become outliers (topic −1) instead of polluting real topics. The number of topics emerges from the data; you never set \(k\).

Step 4 — Describe. Each cluster needs a human-readable label. BERTopic concatenates all documents of a cluster into one pseudo-document and applies class-based TF-IDF:

where \(A\) is the average number of words per class and \(\text{tf}(t)\) is the frequency of \(t\) across all classes. It is the TF-IDF you know applied at the cluster level: words frequent in this topic but rare in others rank highest — those become the topic's keywords.

A basic worked example

BERTopic is not in this site's build environment (embedding models are heavyweight), so run this locally or in Colab — a companion notebook is provided below.

# pip install bertopic
from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups

docs = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes')).data

topic_model = BERTopic(language='english', verbose=True)
topics, probs = topic_model.fit_transform(docs)

topic_model.get_topic_info().head(10)

Typical output — topics discovered with no supervision, no preset \(k\):

Topic  Count  Name
-1     6789   -1_the_of_to_and          ← outliers (no topic)
 0      589   0_game_team_hockey_play
 1      541   1_god_jesus_bible_faith
 2      480   2_car_engine_dealer_miles
 3      432   3_key_encryption_chip_clipper
 ...

Inspecting and using the model:

topic_model.get_topic(0)                     # top c-TF-IDF words of topic 0
topic_model.find_topics("space exploration") # search topics semantically
topic_model.transform(["My car needs new brakes"])  # assign topics to new docs

# Built-in interactive visualizations (Plotly)
topic_model.visualize_topics()      # inter-topic distance map
topic_model.visualize_barchart()    # top words per topic
topic_model.visualize_heatmap()     # topic similarity matrix

Practical tips

• Reduce outliers: a large topic −1 is normal; topic_model.reduce_outliers(docs, topics) reassigns them to the nearest topic if desired.
• Control topic granularity with HDBSCAN's min_cluster_size (via min_topic_size): larger → fewer, broader topics. Or merge after fitting: topic_model.reduce_topics(docs, nr_topics=20).
• Better keywords: pass a CountVectorizer(stop_words='english', ngram_range=(1,2)) to improve c-TF-IDF labels without touching the clustering.
• Reproducibility: UMAP is stochastic — set umap_model=UMAP(random_state=42) for repeatable topics.
• Portuguese / multilingual corpora: BERTopic(language='multilingual') selects a multilingual sentence-transformer — it works well on Brazilian Portuguese text.

LDA vs BERTopic

Companion notebook

Download the notebook and run it in Colab or locally (pip install bertopic):

bertopic_example.ipynb

Quiz