Machine Learning · Insper · E2.4

Sentence Embeddings

Advanced approach

From search to production: grouping sentences, the length limit, cost and the choice of a model

The path of this part

  1. Groupk-means on TF-IDF (term frequency × inverse document frequency) and on embeddings
  2. Mind the limitsnormalize, encode once, the length limit
  3. Cost and choiceparameters, memory, choosing a model, indexes
Each stage answers one question: can sentence vectors group texts with no labels? → what silently goes wrong? → what does it cost, and which model?
Stage 1 · can sentence vectors group texts with no labels?

Clustering sentences

Predict. Headlines of four words share almost no words. Which representation will k-means group better?

16 Portuguese headlines, four subjects (football, cooking, economy, weather), k-means with 4 clusters, scored with the adjusted Rand index (ARI):

TF-IDF

0.02
no better than chance: short headlines share almost no words

embeddings

0.68
football, cooking and weather come out whole
Short texts are where sentence embeddings help clustering most: BERTopic, in the topic modeling lessons, builds a whole method on it.
Stage 2 · what silently goes wrong?

Normalize, encode once, mind the length

  • Normalize (normalize_embeddings=True): the cosine becomes a dot product and a whole similarity matrix is one E @ E.T.
  • Encode in batches, once, save the matrix, and encode only the new texts.

Predict. A 330-token report gets a decisive last sentence. The model reads at most 256 tokens. How much does the embedding change?

tokens 1–256: read by the model
257–330: cut
last sentence
Cosine between the two texts: 1.0000; the last sentence is dropped in silence (the multilingual model reads 128). Split long documents into chunks and embed each.
Stage 3 · what does it cost, and which model?

Cost, choice and scale

cost

22.7 M parameters (117.7 M multilingual): a network forward pass per text, seconds to minutes on a CPU (central processing unit) for thousands of texts. Storage: 384 × 4 bytes = 1.5 KB per text in float32.

choice

Which language? Which kind of text? How much can you pay? Shortlist on MTEB (Massive Text Embedding Benchmark), then measure on your own data.

Scale: for millions of texts, vector databases with approximate nearest-neighbor indexes answer in milliseconds.

Summary

  1. On short texts, k-means on embeddings finds the subjects (ARI 0.68) where TF-IDF finds nothing (0.02); read the clusters before naming them.
  2. Normalize and encode once; a text longer than the model's limit is cut in silence: split it into chunks.
  3. The cost is in computing the vectors; choose the model by language, kind of text and budget, and measure on your data.
Advanced lab · about 25 min, in pairs

Groups, maps and costs, steps 6 to 8

  • Same notebook, same session: the steps use sentences, pairs, ia, ib, E and model.
  • k-means with 8 clusters on the 3,659 sentences, a PCA (principal component analysis) map, and the practical numbers: tokens, memory, parameters.
  • Simulations: is cluster 7 only negations? (step 6) and smaller numbers, same ranking? (step 8).

Pace: step 6 and the first simulation in 10 min; step 7 in 5 min; step 8 and the second in 10 min.