Dimensionality Reduction

Real datasets often have dozens, hundreds, or — for text and images — thousands of features. Dimensionality reduction compresses them into a few informative dimensions, for three reasons:

1. Visualization — humans see in 2D/3D; projecting data reveals clusters, gradients, and outliers;
2. Noise and redundancy removal — correlated features (remember the iris petal measurements) carry duplicated information;
3. The curse of dimensionality — in high dimensions, data becomes sparse and distances lose meaning, degrading distance-based methods like k-NN and clustering.

PCA — Principal Component Analysis

PCA (Pearson, 1901; Hotelling, 1933) is the classical, linear method: find the orthogonal directions of maximum variance and project onto the top few.

The math

Given centered data \(X \in \mathbb{R}^{n \times d}\) (each column has zero mean), the sample covariance matrix is

The first principal component is the unit vector \(w\) maximizing the variance of the projection:

The solution is the eigenvector of \(C\) with the largest eigenvalue \(\lambda_1\); the second component is the next eigenvector, orthogonal to the first, and so on. The eigenvalue \(\lambda_k\) is the variance captured by component \(k\), which gives the explained variance ratio:

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

X_scaled = StandardScaler().fit_transform(X)   # scale first — PCA chases variance!
pca = PCA(n_components=0.95)                   # keep 95% of the variance
Z = pca.fit_transform(X_scaled)
pca.explained_variance_ratio_                  # variance captured per component

Practical notes:

• Components are linear combinations of original features — inspect pca.components_ to interpret them;
• The scree plot (explained variance per component) guides how many components to keep — look for the "elbow";
• PCA is also a compression/denoising tool: reconstruct with few components to filter noise.

Find PC1 by hand — rotate the axis until the projected variance peaks, then check yourself with the snap button:

Nonlinear methods: t-SNE and UMAP

Linear projections cannot unfold curved structures (the classic "Swiss roll"). Two modern nonlinear methods dominate visualization practice:

t-SNE (van der Maaten & Hinton, 2008)

t-SNE converts pairwise distances into neighbor probabilities in high dimension, then finds a 2D layout whose neighbor probabilities match (minimizing KL divergence). It excels at revealing local cluster structure.

Caveats you must know:

• Perplexity (≈ effective number of neighbors, typical 5–50) changes the picture substantially;
• Cluster sizes and inter-cluster distances in a t-SNE plot are not meaningful — the algorithm preserves neighborhoods, not global geometry;
• It is stochastic: different seeds give different layouts;
• There is no transform for new points (in the standard formulation) — it is a visualization tool, not a general feature extractor.

UMAP (McInnes, Healy & Melville, 2018)

UMAP builds a k-nearest-neighbor graph of the data, models its fuzzy topological structure, and optimizes a low-dimensional layout preserving it. Compared with t-SNE it:

• is usually faster and scales better;
• preserves more global structure (relative positions of clusters mean somewhat more);
• supports transform for new points, so it can feed downstream models — this is exactly its role inside BERTopic, where it reduces text embeddings before clustering.

# pip install umap-learn
import umap
Z = umap.UMAP(n_neighbors=15, min_dist=0.1, n_components=2).fit_transform(X_scaled)

PCA vs t-SNE, side by side

Handwritten digits (64 dimensions → 2), same data, two projections:

PCA — the best linear view — overlaps several digit classes: two directions of maximum variance are not enough. t-SNE separates the ten classes almost perfectly by preserving local neighborhoods. The price: axes, cluster sizes and inter-cluster distances in the t-SNE panel have no interpretable meaning.

Choosing a method

Class materials

Video

Latent Space Visualisation: PCA, t-SNE, UMAP

Quiz