Skip to content

Latent Space vs. Embedding

In one sentence

An embedding is the address of one item; a latent space is the whole map those addresses live on β€” including the empty places between them, which a generative model can visit.

In plain words

Think of a map of a city.

  • The embedding is the address of one building: a list of numbers that says where this item is. You use it to ask "what is near this?" β€” the closest buildings are the most similar items. That is search, recommendation, clustering.
  • The latent space is the map itself, every point of it, built by a model. A generative model can stand on any point, even an empty lot no building was ever on, and tell you "what would be here?" That is sampling, interpolation, generation.

Every embedding is a point in some latent space. The two words point at different things: the vector of one input, or the space and what a model can do at every point of it.

The difference in one picture

Two panels. Left, an embedding: an item goes through an encoder to a vector, and the vector is compared with others by cosine to find what is similar. Right, a latent space: the encoder sends data to a space with a prior, and a decoder turns any point of that space, including points no data was sent to, into data. EMBEDDING β€œwhat looks like this?” an item encoder its vector compare by cosine β†’ nearest items search Β· cluster Β· feed a classifier LATENT SPACE β€œwhat is there?” data encoder prior 𝒩(0, I) a point z decoder new data any point, even one no data was sent to sample Β· interpolate Β· denoise Β· plan
Both start with an encoder and both end in a vector space. An embedding is used through its distances; a latent space through what the model can do at every point of it.

Try it

The same two-dimensional space, read two ways. Eight shapes were placed in it; a decoder turns every point into a shape (left to right: square to circle; bottom to top: blue to red).

  • Embedding: click a shape β€” or any empty point, as a new query. The list ranks the others by cosine. Only the direction from the origin matters: no decoder is involved.
  • Latent space: drag the point anywhere. Between two shapes the decoder draws one that was never in the data; far outside the dashed circles (the prior \(\mathcal N(0, I)\)) nothing was trained and the shape falls apart. Interpolate walks a straight line from the red square to the blue circle.
A two-dimensional space with eight shapes placed in it. In embedding mode, choosing a shape ranks the others by cosine similarity. In latent-space mode, a decoder draws the shape that corresponds to any point, and points far from the data come out degraded.

Side by side

Embedding Latent space
Question "what looks like this?" "what is there?"
Points at the vector of one input the space, and what a model does in it
Needs a decoder? no β€” distances are enough usually: to turn a point back into data
Has a distribution? usually not; often normalized to length 1 often a prior you can sample, like \(\mathcal N(0, I)\)
Typical use search, recommendation, clustering, the input of a classifier sampling, interpolation, denoising, planning
In this course token embeddings, CLIP, transfer learning VAE, Stable Diffusion, world models

Three common mistakes

  1. "Latent spaces are big, embeddings are small." Text-embedding models now output 3,072 numbers4, while a world model's latent can be 32. Size does not tell them apart.
  2. "Embeddings are readable, latent spaces are not." Neither has readable coordinates. In both, meaning lives in directions: word2vec's "King βˆ’ Man + Woman β‰ˆ Queen"5, StyleGAN's editing directions, concept directions in LLMs6.
  3. "Latent spaces generate, embeddings only represent." A JEPA plans in a latent space without generating a pixel, and Stable Diffusion generates from CLIP's text embeddings7.

Where the two words meet

  • An LLM's hidden state is both a contextual embedding of a token and the latent the next block reads.
  • A VAE encoder's mean \(\mu(x)\) is a latent code; used for search, it is an embedding.
  • CLIP's embedding space is the latent that Stable Diffusion's text conditioning lives in.

The formal definitions

Latent space. The space of a model's latent β€” unobserved β€” variables. A latent-variable model explains the data through a hidden code, \(p(x) = \int p(x \mid z)\,p(z)\,dz\); \(z\) lives in the latent space, usually with a prior such as \(\mathcal N(0, I)\) in a VAE1. The word is also used for any learned internal representation. A latent space can be much smaller than the data because of the manifold hypothesis: real data concentrate near a manifold of much lower dimension than the space they are written in2.

Embedding. "A vector representation of data in embedding space", found by projecting high-dimensional data into a lower-dimensional space where similar items land close together3. The word names the learned map (an embedding layer, an embedding model) and the vector it returns. What makes a vector an embedding is that it was learned and that distance in it is meant to be used.

Go deeper

  • Latent Space β€” the latent of each chapter, generative latents, and VAE, JEPA, world models and TDWM up close.
  • Embedding β€” from a token id to a searchable vector, the geometry of the space, text-embedding models and what an embedding gives away.

  1. Kingma, D. P., & Welling, M. (2014). Auto-Encoding Variational Bayes β€” ICLR. ↩

  2. Bengio, Y., Courville, A., & Vincent, P. (2013). Representation Learning: A Review and New Perspectives β€” IEEE TPAMI 35(8). ↩

  3. Google for Developers. Machine Learning Crash Course β€” Embeddings: embedding space and static embeddings. ↩

  4. OpenAI (2024, January 25). New embedding models and API updates β€” text-embedding-3-large. Kusupati, A., et al. (2022). Matryoshka Representation Learning β€” NeurIPS. ↩

  5. Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations β€” NAACL-HLT. ↩

  6. Park, K., Choe, Y. J., & Veitch, V. (2024). The Linear Representation Hypothesis and the Geometry of Large Language Models β€” ICML. ↩

  7. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-Resolution Image Synthesis with Latent Diffusion Models β€” CVPR. ↩