Most tasks are about texts, not words
support
find the FAQ entry that answers a question
operations
group support tickets
search
retrieve the documents that match a query
Goal: one vector per sentence or short document, arranged so that the cosine between two of them tracks how close their meanings are.
The path of this lesson
- Average the wordsmean of GloVe vectors; order and negation vanish
- Words in contextTransformers: one word, several vectors
- One vector per sentenceSentence-BERT against TF-IDF and averaging; other languages
- Search and groupsemantic search, clustering, practical notes
Each stage answers one question: is averaging the word vectors enough? → how can one word get several vectors? → how do we get one vector that compares sentences? → what do we do with sentence vectors?
Stage 1 · Is averaging the word vectors enough?
The obvious answer: average the word vectors
| "the movie was great" | first four of the 50 GloVe numbers |
|---|---|
| the | 0.418 · 0.25 · −0.412 · 0.122 … |
| movie | 0.308 · 0.172 · −0.233 · 0.023 … |
| was | 0.087 · −0.194 · −0.243 · −0.334 … |
| great | −0.027 · 1.336 · −1.028 · −0.373 … |
| mean | 0.197 · 0.391 · −0.479 · −0.141 … |
By hand: \((0.418 + 0.308 + 0.087 - 0.027)/4 = 0.1965\). Costs nothing, needs no training.
What averaging loses
Predict. "the dog bit the man" vs "the man bit the dog": what cosine do their averages have?
| 1.00 | the dog bit the man / the man bit the dog |
| 0.99 | the movie was good / the movie was not good |
| 0.96 | the movie was great / the film was excellent |
| 0.66 | the stock market fell sharply today / my cat likes to sleep on the couch |
order is gone
same words, same mean: exactly 1.00
negation barely registers
"not" is one word in five
everything looks similar
unrelated sentences still score 0.66
Stage 2 · How can one word get several vectors?
Transformers, in one slide
- A Transformer (Vaswani et al., 2017) reads the whole sentence at once.
- It cuts the text into subword tokens (a vocabulary of about 30,000 pieces): no word is ever out of vocabulary.
- In each layer, every token's vector is recomputed as a weighted mix of all the tokens (attention).
- BERT (Devlin et al., 2019): trained to fill in hidden words; one contextual vector per token.
Embeddings are unbelievably useful → em · ##bed · ##ding · ##s · are · un · ##bel · ##ie · ##va · ##bly · useful
Here it is a black box that turns text into vectors; how it is built is the subject of Neural Networks.
One word, several vectors
Predict. The token "bank" in three river sentences and three money sentences. In GloVe all 36 cells would be 1.00. Here?
| river 1 | river 2 | river 3 | money 4 | money 5 | money 6 | |
|---|---|---|---|---|---|---|
| river 1 | 1.00 | 0.83 | 0.86 | 0.76 | 0.76 | 0.76 |
| river 2 | 0.83 | 1.00 | 0.84 | 0.72 | 0.72 | 0.73 |
| money 4 | 0.76 | 0.72 | 0.72 | 1.00 | 0.88 | 0.86 |
| money 5 | 0.76 | 0.72 | 0.73 | 0.88 | 1.00 | 0.82 |
Every river "bank" is closer to the other river "banks" (0.83 to 0.86) than to any money "bank" (0.71 to 0.76). That is what contextual means.
Stage 3 · How do we get one vector that compares sentences?
Sentence-BERT: trained to compare sentences
Mean pooling: average the token vectors, each already carrying its context.
Sentence-BERT (Reimers & Gurevych, 2019) adds training: cosine up for pairs that mean the same, down for pairs that do not.
A bi-encoder: each sentence is encoded separately, so a collection is encoded once and stored.
Using sentence-transformers
E = model.encode(["the movie was great", "the film was excellent",
"the stock market fell sharply today"], normalize_embeddings=True)
E @ E.T # with unit rows, the dot product is the cosine
size
384
numbers per text
paraphrases
0.88
the stock market sentence: 0.04
the model
22.7 M
parameters · reads at most 256 tokens
Averaged GloVe gave 0.96 against 0.66: now the scale is used from bottom to top.
Three representations, the same pairs
TF-IDF, averaged GloVe vectors and Sentence-BERT (SBERT) on eight pairs, each probing one property:
| pair | TF-IDF | avg GloVe | SBERT |
|---|---|---|---|
| paraphrase | 0.28 | 0.96 | 0.88 |
| paraphrase, other words | 0.19 | 0.92 | 0.56 |
| same need, other words | 0.15 | 0.92 | 0.68 |
| negation | 0.82 | 0.99 | 0.82 |
| word order | 1.00 | 1.00 | 0.98 |
| bank: river vs money | 0.29 | 0.92 | 0.60 |
| apple: company vs fruit | 0.18 | 0.87 | 0.23 |
| unrelated | 0.04 | 0.66 | 0.02 |
Before reading the next slide: which column would you trust to separate related from unrelated?
Reading the table
TF-IDF
measures shared words: paraphrases without them score 0.15 to 0.28
avg GloVe
says "similar" to everything: 0.66 to 1.00, no threshold works
SBERT
uses the whole scale: 0.02 to 0.88; the two "apples" score 0.23
What none of them fixes: negation stays at 0.82 and word order at 0.98. If your task depends on them (sentiment, contradictions, who did what), check it explicitly.
Other languages
| pair (Portuguese) | English-only | multilingual |
|---|---|---|
| o filme foi ótimo / o longa foi excelente | 0.65 | 0.66 |
| esqueci minha senha / não lembro a senha da conta | 0.40 | 0.81 |
| o filme foi ótimo / the movie was great | 0.08 | 0.98 |
| o filme foi ótimo / a bolsa de valores caiu hoje | 0.40 | −0.02 |
The English model does not fail loudly: it gives a paraphrase and an unrelated pair the same 0.40. Choose a model trained on your language.
Stage 4 · What do we do with sentence vectors?
Semantic search is k-nearest neighbors
Encode the collection once; for each query, one matrix product gives every cosine. No labels, no prediction: only finding neighbors.
An FAQ in Portuguese, eight entries
| query | 1st answer | cos |
|---|---|---|
| esqueci minha senha | Como faço para trocar a senha da minha conta? | 0.68 |
| onde está minha encomenda? | Como rastrear o meu pedido? | 0.60 |
| desisti da compra | Como cancelar uma compra que ainda não foi enviada? | 0.62 |
| o celular veio com defeito | Posso devolver um produto depois de usado? (warranty 2nd, 0.24) | 0.26 |
| I need a copy of my invoice | Como faço para trocar a senha… (miss) | 0.43 |
Four of six right with different words. Low, close scores mean "unsure"; idioms ("segunda via da nota fiscal") are where general embeddings fail.
Simulator: embeddings vs TF-IDF
Predict. For the query "erro 402", which ranking wins, embeddings or TF-IDF? Then type your own query.
What the simulator shows
embeddings
15 / 19
relevant sentences in the top 3, over ten queries
TF-IDF
6 / 19
but it wins "erro 402": the rare token is exact
Production search engines combine both rankings: hybrid search.
Clustering sentences
16 Portuguese headlines, four subjects (football, cooking, economy, weather), k-means with 4 clusters, scored with the adjusted Rand index (ARI):
TF-IDF
0.02
no better than chance: short headlines share almost no words
embeddings
0.68
football, cooking and weather come out whole
Short texts are where sentence embeddings help clustering most: BERTopic (U5.3) builds a whole method on it.
Practical notes
- Normalize: cosine becomes a dot product.
- Encode in batches, once, and save the matrix.
- Length limit: a 330-token text plus a decisive last sentence → cosine 1.0000 (the model reads 256; the multilingual one 128). Split long documents into chunks.
- Cost: 22.7 M parameters (117.7 M multilingual), seconds to minutes on a CPU; 1.5 KB per text in float32.
- Choose: language, kind of text, budget; shortlist on MTEB, measure on your data.
- Scale: vector databases for millions of texts.
Summary
- Averaging word vectors: free, but blind to order and negation, and everything looks alike.
- A Transformer gives each token a contextual vector: two "banks", two vectors.
- Sentence-BERT: pooled and trained so that the cosine tracks meaning, over the whole scale.
- Semantic search = nearest neighbors; clustering short texts works much better than on TF-IDF.
- Check negation and roles, choose the model for your language, mind the length limit.
Lab · about a minute on a laptop CPU after the downloads
Sentence similarity in Portuguese
- ASSIN 2: 2,448 test pairs of Brazilian Portuguese sentences rated by people from 1 to 5.
- Score them with TF-IDF and two sentence-transformer models, measure how well each follows the human score, read where they disagree.
- Then search, cluster and draw the 3,659 distinct sentences.
- Download the lab notebook and open it in Jupyter or Colab; in Colab run
!pip install sentence-transformersfirst.
Model downloads: about 90 MB and 470 MB; a Colab GPU makes the lab a few seconds.