Dimensionality Reduction

Real datasets often have dozens, hundreds, or — for text and images — thousands of features. Dimensionality reduction compresses them into a few informative dimensions, for three reasons:

1. Visualization — humans see in 2D/3D; projecting data reveals clusters, gradients, and outliers;
2. Noise and redundancy removal — correlated features (remember the iris petal measurements) carry duplicated information;
3. The curse of dimensionality — in high dimensions, data becomes sparse and distances lose meaning, degrading distance-based methods like k-NN and clustering.

The curse of dimensionality, measured

Reason 3 above is usually asserted and rarely shown, so start there. In high dimensions something counter-intuitive happens to distance itself: every pair of points ends up roughly equally far apart.

Take n points uniformly at random in the unit cube and look at all pairwise distances. In two dimensions the histogram is wide — some pairs are close, others far, and "nearest neighbour" picks out something meaningful. Push the dimension up and the histogram collapses into a narrow spike:

The number to watch is the relative contrast, \((d_{\max} - d_{\min})/d_{\min}\). At \(d = 2\) the farthest point is typically tens of times farther than the nearest. By \(d = 256\) that ratio falls below 1.5 — the nearest neighbour is barely nearer than the farthest one.

Everything that relies on distance degrades with it: k-NN, k-means, RBF kernels, and t-SNE and UMAP themselves. This is why those two methods recommend running PCA first when the input has hundreds of dimensions. It is not only about speed — it is about feeding them neighbourhoods that still mean something.

PCA — Principal Component Analysis

PCA (Pearson, 1901; Hotelling, 1933) is the classical, linear method: find the orthogonal directions of maximum variance and project onto the top few.

The math

Given centered data \(X \in \mathbb{R}^{n \times d}\) (each column has zero mean), the sample covariance matrix is

The first principal component is the unit vector \(w\) maximizing the variance of the projection:

The solution is the eigenvector of \(C\) with the largest eigenvalue \(\lambda_1\); the second component is the next eigenvector, orthogonal to the first, and so on. The eigenvalue \(\lambda_k\) is the variance captured by component \(k\), which gives the explained variance ratio:

Worked by hand

Five points, chosen so every number comes out whole:

The mean is \((5, 5)\). With \(n - 1 = 4\):

Now solve \(\det(C - \lambda I) = 0\), which for a 2×2 is a quadratic you can do in your head:

Two checks worth internalising: the eigenvalues sum to the trace (8 + 2 = 10 = 5 + 5), and the trace is the total variance. So \(\text{EVR}_1 = 8/10 = 80\%\).

Substituting \(\lambda_1 = 8\) into \((C - 8I)w = 0\) gives \(-3w_1 + 3w_2 = 0\), so \(w_1 = w_2\): PC1 is \((1,1)/\sqrt{2}\) — exactly the 45° diagonal, which is where the cloud visibly points. PC2 is forced to be \((1,-1)/\sqrt{2}\), perpendicular to it.

Projecting gives PC1 scores \(-2\sqrt2,\ -2\sqrt2,\ 0,\ 2\sqrt2,\ 2\sqrt2\), whose variance is 8 — the eigenvalue again. The eigenvalue is not a proxy for the variance along the component; it is that variance.

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

X_scaled = StandardScaler().fit_transform(X)   # scale first — PCA chases variance!
pca = PCA(n_components=0.95)                   # keep 95% of the variance
Z = pca.fit_transform(X_scaled)
pca.explained_variance_ratio_                  # variance captured per component

Practical notes:

• Components are linear combinations of original features — inspect pca.components_ to interpret them;
• The scree plot (explained variance per component) guides how many components to keep — look for the "elbow";
• PCA is also a compression/denoising tool: reconstruct with few components to filter noise.

That derivation says PCA "maximises projected variance", which sounds abstract until you do it by hand. Rotate the line yourself: the bar on the right is the variance captured at that angle, and the grey segments are what gets thrown away. There is exactly one angle where the bar peaks — press find PC1 to snap to it and confirm it is the one you were converging on.

Reconstruction: what k components actually keep

PCA is reversible. Drop from \(d\) dimensions to \(k\) and project back, and you get an approximation whose error is exactly the variance you discarded — \(\sum_{j>k}\lambda_j\). On the five points above, keeping only PC1 discards \(\lambda_2 = 2\), i.e. 20% of the variance.

The simulator below does this on 12×12 images. Slide \(k\) and watch the reconstruction: the interesting part is how few components the eye needs before it stops objecting.

Practically, this is why PCA doubles as compression and denoising: the discarded directions are the low-variance ones, and noise tends to live there.

Where PCA runs out

PCA can only rotate and project — it moves the axes, never bends them. When the structure in the data is a bend, that is fatal.

The Swiss roll is the standard counter-example: a 2-D sheet rolled up in 3-D. Two points on opposite layers of the roll can be close in straight-line distance while being very far apart along the sheet. Tick mark a pair of points to see one such pair, then project with PCA and watch the two colours smear into each other:

The distinction the simulator makes visible is between Euclidean distance (through the empty space between layers) and geodesic distance (walking along the surface). Manifold methods are the ones that use the second.

Nonlinear methods: t-SNE and UMAP

Two modern nonlinear methods dominate visualization practice:

t-SNE (van der Maaten & Hinton, 2008)

t-SNE converts pairwise distances into neighbor probabilities in high dimension, then finds a 2D layout whose neighbor probabilities match (minimizing KL divergence). It excels at revealing local cluster structure.

Caveats you must know:

• Perplexity (≈ effective number of neighbors, typical 5–50) changes the picture substantially;
• Cluster sizes and inter-cluster distances in a t-SNE plot are not meaningful — the algorithm preserves neighborhoods, not global geometry;
• It is stochastic: different seeds give different layouts;
• There is no transform for new points (in the standard formulation) — it is a visualization tool, not a general feature extractor.

Those caveats are easier to believe once you have watched them happen. The simulator below runs real gradient descent on the KL divergence, in your browser — nothing is pre-computed:

Three experiments worth doing before you trust any t-SNE plot you did not make yourself:

1. pick uniform noise and run it. t-SNE will still produce tidy-looking blobs. The method always outputs clusters; the data does not always contain them.
2. keep the dataset and change only the perplexity. The number of apparent groups can change. Report the perplexity alongside any t-SNE figure, the way you would report a random seed;
3. pick groups of different sizes and compare the on-screen sizes with the real ones. They do not match — t-SNE equalises dense and sparse regions, which is exactly why reading cluster size or inter-cluster distance off a t-SNE plot is a mistake.

UMAP (McInnes, Healy & Melville, 2018)

UMAP builds a k-nearest-neighbor graph of the data, models its fuzzy topological structure, and optimizes a low-dimensional layout preserving it. Compared with t-SNE it:

• is usually faster and scales better;
• preserves more global structure (relative positions of clusters mean somewhat more);
• supports transform for new points, so it can feed downstream models — this is exactly its role inside BERTopic, where it reduces text embeddings before clustering.

# pip install umap-learn
import umap
Z = umap.UMAP(n_neighbors=15, min_dist=0.1, n_components=2).fit_transform(X_scaled)

PCA vs t-SNE, side by side

Handwritten digits (64 dimensions → 2), same data, two projections:

PCA — the best linear view — overlaps several digit classes: two directions of maximum variance are not enough. t-SNE separates the ten classes almost perfectly by preserving local neighborhoods. The price: axes, cluster sizes and inter-cluster distances in the t-SNE panel have no interpretable meaning.

Choosing a method

Class materials

Video

Latent Space Visualisation: PCA, t-SNE, UMAP

References

• Pearson, K. "On Lines and Planes of Closest Fit to Systems of Points in Space." Philosophical Magazine 2 (1901). DOI
• Hotelling, H. "Analysis of a Complex of Statistical Variables into Principal Components." J. Educational Psychology 24 (1933). DOI
• van der Maaten, L.; Hinton, G. "Visualizing Data using t-SNE." JMLR 9 (2008). full text
• McInnes, L.; Healy, J.; Melville, J. "UMAP: Uniform Manifold Approximation and Projection." arXiv (2018). arXiv:1802.03426

The full course bibliography is on the references page.

Quiz