Skip to content

Embedding

An embedding is a learned map from things that have no geometry of their own — token ids, categories, users, images, sentences — to vectors whose geometry means something, and, by extension, the vector that map returns. Google's crash course puts it in one line: "an embedding is a vector representation of data in embedding space", found by projecting high-dimensional data into a lower-dimensional space so that similar items land close together1. Two things make a vector an embedding rather than any list of numbers: it was learned, and distance in it is meant to be used — for similarity, retrieval, or as the input of the next model.

The word covers three objects that the course meets in this order:

What it is Where in the course
A lookup embedding one learned row per discrete symbol: a table \(E \in \mathbb R^{V \times d}\) feature types, the embedding layer
A contextual embedding the vector of a token after the network has read its neighbours attention, the block
A pooled embedding one vector for a whole input — a sentence, an image — from an encoder trained so that cosine means similarity CLIP, transfer learning

Every embedding lives in a latent space; the latent-space page looks at those spaces from the other side, starting from the models that build them.

From an id to a searchable vector

From a token id to a searchable vector, in seven stations: the id, a one-hot vector, a row of the embedding table, the position added, the contextual vector after the Transformer, one pooled and normalized vector per sentence, and a search by cosine in the embedding space. 1 · IDa name 3797“␣cat” 2 · ONE-HOT50,257 slots … 3 · LOOKUPa row of E 768 4 · + POSITIONx = E[i] + P E[i] + P x 5 · CONTEXTafter the blocks “river bank” “bank loan” 6 · POOLone vector, ‖v‖ = 1 7 · SEARCHnearest by cosine query MATRYOSHKA 3072 → 256

The embeddings of each chapter

Chapter The embedding Size in the course
Data · feature types a learned vector per category, instead of one-hot 40,000 SKUs → 32
Attention · tokens none yet: an id is a name, "4 is not twice 2" GPT-2: 50,257 ids
Attention · embeddings the table \(E\), plus the position \(P\) 50,257 × 768 and 1024 × 768 in GPT-2
Transformers · block contextual: each block adds a correction to every row \(n \times d\) in every block
Transformers · output the unembedding: one direction per token, often tied to \(E\) 768 × 50,257
Vision Transformers a patch embedding: \(P^2C\) pixels → \(d\) 768 → 768 in ViT-B/16
Transfer Learning a frozen encoder's output, read by a linear probe 768 in DistilBERT
CLIP one unit vector per image and per caption, in a shared space 512 for ViT-B/32
Stable Diffusion the prompt's token embeddings, read by cross-attention 77 × 768
Evaluation of generative models BERTScore, MAUVE and CLIPScore compare in embedding space —
Evaluation of LLMs the retrieval half of a RAG system —
JEPA embeddings as the target of a prediction one vector per patch

All the techniques on one map

Every technique this page mentions, in five families. Click a box for what it is, with a link to the chapter or to the section of this page that treats it; Next follows seven threads, from a one-hot vector to what an embedding gives away.

A map of every embedding technique of the page, in five families, with the links between them.

How a lookup embedding learns

The embedding layer is the cheapest layer to run and the most unusual to train. Multiplying a one-hot by \(E\) only picks a row, so the gradient reaches only the rows of the tokens in the batch: in a step over a few thousand tokens most rows receive exactly zero. Frequent tokens are updated at every step, rare ones a few times per epoch, and a rare token learns slowly. For tabular categories the feature-types page adds a practical note: the published rules for the dimension (\(k^{1/4}\), \(1.6\,k^{0.56}\)) disagree with each other, so start near 16 or 32 and tune it.

The geometry of an embedding space

  • Cosine is the distance. On unit vectors \(\|a - b\|^2 = 2 - 2\cos(a, b)\), so nearest neighbour and highest cosine are the same search. That is why CLIP and every retrieval encoder normalize.
  • Directions are cheap. Two random directions in \(d\) dimensions have a cosine of mean zero and spread \(1/\sqrt d\); at \(d = 1000\), 99% of pairs have \(|\cos| < 0.082\) (A4 · Everything becomes orthogonal). A network can therefore store more features than dimensions, each as an almost orthogonal direction — superposition2. Sparse autoencoders trained on the residual stream of a production LLM recover millions of such directions, about 34 million in the largest run3.
  • Concepts are (often) directions. Word2vec showed that relations are vector offsets: "King − Man + Woman" lands close to "Queen", and the offsets answered almost 40% of a set of syntactic analogy questions4. The linear representation hypothesis makes this precise for LLMs and connects it to the two tools the course uses: linear probes read a concept, adding the direction steers the model5.
  • A trained table has no readable columns. What one coordinate means is spread over many of them; meaning lives in directions and distances, not in axes.

Text embeddings for search

Outside this course, "embedding" most often means a text-embedding model: an encoder that turns a whole sentence or document into one vector so that a search is a nearest-neighbour lookup.

Year Model What it changed
2019 Sentence-BERT6 a siamese BERT whose sentence vectors can be compared by cosine, instead of running BERT on every pair
2020 DPR7 a dual encoder for questions and passages that beat BM25 keyword search by 9 to 19 points of top-20 accuracy
2022 E58 contrastive pretraining on weakly paired text; the first model to beat BM25 zero-shot on BEIR
2022 Matryoshka9 the useful information is packed in the first coordinates, so a vector can be truncated
2024 text-embedding-310 3,072 numbers that can be cut to 256 and still beat the previous model's 1,536 on MTEB

They are compared on MTEB11, and since 2025 on MMTEB, more than 500 tasks in more than 1,000 languages; there the best public model was a 560-million-parameter encoder, not an LLM12. In a retrieval-augmented system this is the retrieval half, which is why the LLM evaluation page measures it on its own.

What an embedding gives away

An embedding is not an anonymised summary. Vec2Text reconstructs 92% of 32-token Wikipedia sentences exactly from their embeddings13: storing vectors instead of text in a vector database is not, by itself, a privacy measure. In the other direction, different models are converging on similar geometries — as vision and language models grow, they measure distances between data points in more and more alike ways — which is what the platonic representation hypothesis proposes to explain14.

Key takeaways

  1. An embedding is a learned vector whose distances are meant to be used. The word names the map and the vector.
  2. Three kinds: a lookup row per symbol, a contextual vector after the network has read the neighbours, a pooled vector per input. The course meets them in that order.
  3. A lookup table learns through a sparse gradient: only the rows in the batch move, and rare symbols learn slowly.
  4. The geometry is cosine on the sphere; directions are cheap, so models store more features than dimensions, and many concepts are directions.
  5. Text-embedding models turn search into geometry; MTEB measures them, Matryoshka training lets them be cut.
  6. An embedding leaks most of its input: it can be inverted.

  1. Google for Developers. Machine Learning Crash Course — Embeddings: embedding space and static embeddings. ↩

  2. Elhage, N., et al. (2022). Toy Models of Superposition — Transformer Circuits Thread. ↩

  3. Templeton, A., Conerly, T., et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet — Transformer Circuits Thread. ↩

  4. Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations — NAACL-HLT. ↩

  5. Park, K., Choe, Y. J., & Veitch, V. (2024). The Linear Representation Hypothesis and the Geometry of Large Language Models — ICML. ↩

  6. Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — EMNLP-IJCNLP. ↩

  7. Karpukhin, V., et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering — EMNLP. ↩

  8. Wang, L., et al. (2022). Text Embeddings by Weakly-Supervised Contrastive Pre-training — E5. ↩

  9. Kusupati, A., et al. (2022). Matryoshka Representation Learning — NeurIPS. ↩

  10. OpenAI (2024, January 25). New embedding models and API updates — text-embedding-3-small and -large, with a dimensions parameter. See also Pinecone's summary. ↩

  11. Muennighoff, N., Tazi, N., Magne, L., & Reimers, N. (2023). MTEB: Massive Text Embedding Benchmark — EACL. ↩

  12. Enevoldsen, K., et al. (2025). MMTEB: Massive Multilingual Text Embedding Benchmark — ICLR. ↩

  13. Morris, J. X., Kuleshov, V., Shmatikov, V., & Rush, A. M. (2023). Text Embeddings Reveal (Almost) As Much As Text — EMNLP. ↩

  14. Huh, M., Cheung, B., Wang, T., & Isola, P. (2024). The Platonic Representation Hypothesis — ICML. ↩