8.4. LLM

Every other page in this chapter evaluates a model whose output can be checked. A class is right or wrong; a number is close or far. A language model's output is a paragraph, and there is no key to compare it against — there are many good answers, many bad ones, and no function that separates them.

So every metric here is a proxy, and the useful question about each one is not "how do I compute it" but what does it stand in for, and when does the substitution break? That is the organizing question of this page, because in evaluation of language models the substitution breaks constantly, and quietly.

Perplexity: how surprised the model is

The oldest metric, and the only one computed directly from what the model was trained to do. Over a held-out text of \(N\) tokens:

\[ \text{PPL}(W) = \exp\!\left(-\frac{1}{N}\sum_{i=1}^{N} \log p_\theta(w_i \mid w_{<i})\right) \]

It is the exponential of the average cross-entropy, so it has a clean reading: a perplexity of 10 means the model was, on average, as uncertain as if it had been choosing uniformly among 10 options at every token. Lower is better; a model that spreads its probability uniformly over a vocabulary of \(V\) tokens scores exactly \(V\).

how surprised the model is by Paris p = 0.72 tokenize into subwords

Two properties of that formula do most of the damage in practice, and both are visible above:

It is a geometric mean of \(1/p\). Drag the probability of "Paris" towards zero: one token the model finds impossible drags the whole number up, no matter how well the rest went. Perplexity reports the tokens a model finds hardest.
It divides by the number of tokens. Tick the subword box: the same sentence, the same model, the same total log-probability — and a different perplexity, because the denominator changed.

Perplexity is not comparable across models unless the tokenizer is identical

This is the most common misreading of a published number. A model with a larger vocabulary splits text into fewer tokens, and the per-token average falls without the model being any better at predicting text. The quantity that survives re-tokenization is the total log probability of the corpus — bits per byte or bits per character — which is why careful comparisons report those instead¹.

And it does not measure whether the model is useful

Perplexity is a measure of fit to a corpus. It says nothing about following instructions, reasoning, refusing harmful requests, or being right. A model fine-tuned to be helpful usually gets worse perplexity on raw web text than the base model it came from — the two are simply not measuring the same thing.

Overlap metrics, and why they are the wrong tool here

BLEU and ROUGE compare the generated string to a reference string by counting shared n-grams. They are covered with a simulator in the generative metrics page, and the short version is: they measure surface overlap, not meaning. A correct paraphrase scores badly; a fluent, wrong answer that reuses the reference's vocabulary scores well.

Metric	Counts	Still used for
BLEU	n-gram precision, with a brevity penalty	Machine translation, where references are tight
ROUGE-1/2/L	n-gram recall, and longest common subsequence	Summarization, mostly by convention
BERTScore²	cosine similarity between contextual embeddings, matched greedily	When paraphrase must not be punished

They survive because they are cheap and reproducible, not because they are good. For open-ended generation — which is most of what a language model does — they correlate weakly with human judgement, and the field has moved to the two approaches below.

Benchmarks: measuring tasks instead of fluency

The modern answer is to stop scoring the text and score the task: give the model problems with checkable answers and count how many it gets right.

Benchmark	What it asks	Format	The catch
MMLU³	57 subjects, undergraduate to professional level	Multiple choice	Sensitive to prompt format and answer-order; scores move several points on formatting alone
HumanEval⁴	164 Python functions from a docstring	Write code, run tests	Small, and heavily contaminated by now
GSM8K⁵	8.5k grade-school word problems	Chain of thought, exact match	Answer-only matching hides reasoning that got there by luck
HellaSwag⁶	Commonsense sentence completion	Multiple choice	Humans get ~95%; the residual is mostly noise in the items
TruthfulQA⁷	Questions where the common answer is false	Free generation	Measures resistance to popular misconceptions, not truthfulness in general
MT-Bench⁸	80 multi-turn instructions	Scored 1–10 by an LLM judge	Inherits every bias of the judge — see below
Chatbot Arena⁹	Anonymous A/B votes from real users	Elo rating	The closest thing to a real preference measure; slow, and shifts with the user population

That last row is the only one whose number is not a percentage, so it is worth saying what it is. An Elo rating turns a pile of pairwise votes into one number per model, by assuming the probability that A beats B depends only on the gap between their ratings:

\[ \mathbb{P}(A \text{ beats } B) = \frac{1}{1 + 10^{\,(R_B - R_A)/400}} \qquad\qquad R_A \leftarrow R_A + K\,(S_A - \mathbb{E}[S_A]) \]

The 400 is a convention, and it is what fixes the scale: a 400-point gap means the stronger model is expected to win 10 times out of 11. \(S_A\) is the actual outcome (1, 0 or ½) and \(K\) sets how fast a rating moves. Two consequences worth carrying: the number is relative, so it is meaningless without the population it was computed against, and it measures preference, which is not the same thing as correctness — a model that writes longer, friendlier, better-formatted wrong answers climbs.

Contamination: the benchmark is in the training data

Every public benchmark eventually ends up in the crawl that trains the next model, and a model that has memorized the test set scores well without being able to do the task. This is measured, not hypothetical¹⁰. Consequences you should carry into any benchmark table you read:

A score is only evidence if the benchmark is newer than the model's training cutoff, or held out privately.
Improvements of one or two points on a saturated benchmark are usually not real.
This is why held-out, rotating, or private evaluations — and human preference arenas — have displaced static benchmarks at the top of the field.

pass@k, and why the k is half the number

For code, the natural metric is: does it pass the tests? But a model samples, so its score depends on how many attempts it is allowed. The Codex paper⁴ defines the estimator everyone uses: draw \(n\) samples per problem, count the \(c\) that pass, and estimate the probability that at least one of \(k\) attempts succeeds:

\[ \text{pass@}k = \mathbb{E}_{\text{problems}}\left[1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}\right] \]

model strength 1.00 k — attempts you are allowed 1

The original paper makes the point better than any argument: Codex solves 28.8% of HumanEval at pass@1 and 70.2% at pass@100⁴. Same model, same problems, a difference of forty points that is entirely about how many tries you count.

Which k is honest depends on what you are building

pass@1 is the number for anything that has to be right the first time — a completion that ships, an agent step nobody reviews. pass@k with large k is the right number when a cheap, reliable verifier picks the winner: a test suite, a compiler, a proof checker. Reporting a large \(k\) without a verifier is reporting a score no user will ever experience.

LLM-as-a-judge

Since no formula scores an open-ended answer, the practical answer is to ask a strong model. It scales, it is cheap, and it agrees with human annotators more than you would expect: over 80%, which is about the level at which two humans agree with each other⁸.

That number is also the trap, because the disagreements are not random — they are biased in ways that make a model look better or worse for reasons that have nothing to do with the answer:

how much better B really is 55% position bias (prefers what it reads first) 0.6 verbosity bias (B is the longer answer) 0.4

Bias	What it does	What to do about it
Position	Prefers whichever answer it reads first	Judge every pair twice, in both orders, and average. Report the agreement rate
Verbosity	Prefers the longer answer, at equal quality	Control for length; compare answers of similar length, or report length alongside the win rate
Self-enhancement	Prefers text from its own model family	Never let a model be the judge of its own outputs in an evaluation you will publish
Scale compression	Clusters scores in 7–9 on a 1–10 scale	Prefer pairwise comparisons to absolute scores

A judge is a model, so evaluate it before you trust it

The only way to know your judge works is to check it against human labels on a sample of your own data — a few hundred pairs is usually enough to estimate agreement. Report that agreement number next to every result the judge produced. A win rate without it is a measurement with no stated error.

RAG: two systems, two failure modes

A retrieval-augmented system can fail by retrieving the wrong documents or by ignoring the right ones, and a single end-to-end score cannot tell you which. So they are measured separately¹¹:

Stage	Metric	Question it answers
Retrieval	Context precision	Of what was retrieved, how much was relevant?
	Context recall	Of what was relevant, how much was retrieved?
Generation	Faithfulness	Is every claim in the answer supported by the retrieved context?
	Answer relevancy	Does the answer address the question that was asked?

The pair that matters most is faithfulness against answer relevancy: an answer can be perfectly grounded in the context and useless, or fluent, relevant and entirely invented. Grounding is the one a RAG system was built to provide, and the one users cannot check for themselves.

Safety and alignment

What you are measuring	Typical instrument	What it misses
Toxicity	Classifier over generations (e.g. Perspective API)	Implicit harm, and it inherits the classifier's own biases
Hallucination rate	TruthfulQA, or claim-checking against sources	Only covers the domain you tested
Instruction-following	Verifiable constraints ("answer in 3 bullets, no more")	Says nothing about answer quality
Refusal	Rate on a red-team set, plus over-refusal on benign look-alikes	One rate alone is gameable: a model that refuses everything scores perfectly
Calibration	Confidence against observed accuracy — see classification	A well-calibrated model can still be uniformly wrong

Always measure the pair, never the single rate

Refusal rate without over-refusal rate, precision without recall, faithfulness without relevancy: every safety metric has a partner that a degenerate model can exploit. return "I cannot help with that" is a perfect safety model on any one-sided measurement.

Which metric, and why

Your question	Use	Do not use
Is the language model learning to model this corpus?	Perplexity / bits per byte, same tokenizer	Perplexity across tokenizers
Does it know a subject?	MMLU-style benchmarks, ideally uncontaminated	Anything from before the training cutoff, uncritically
Can it write code that works?	pass@1 for one-shot use; pass@k with a verifier	pass@k compared across different k
Is answer A better than answer B?	Pairwise judge, both orders, with measured human agreement	A single-order win rate; a 1–10 absolute score
Is the summary faithful to the source?	Faithfulness / claim grounding	ROUGE
Is it safe?	Paired rates: refusal and over-refusal	Any single rate
Do users prefer it?	A/B with real users; Arena-style Elo	Benchmark averages

Key takeaways

Every metric on this page is a proxy. Know what it stands in for, and you will know where it breaks.
Perplexity measures fit to a corpus. It is a geometric mean, so the hardest tokens dominate, and it is not comparable across tokenizers — report bits per byte if you must compare.
BLEU/ROUGE measure string overlap. For open-ended generation they are a convention, not a measurement.
Benchmarks measure tasks, and their central risk is contamination: a number is only evidence if the test could not have been trained on.
pass@k without its \(k\) is meaningless — 28.8% and 70.2% were the same model⁴. Large \(k\) is honest only when a verifier picks the winner.
LLM judges agree with humans about 80% of the time⁸, with position, verbosity and self-preference biases. Judge both orders, report agreement, never let a model grade itself.
RAG needs retrieval and generation measured separately, and faithfulness is the one the user cannot verify alone.
Safety metrics come in pairs. A single rate is always gameable.

Additional Resources

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng, L., et al. (2023)⁸. The paper that made LLM judging respectable and documented what is wrong with it. Read sections 3 and 4 together: the agreement numbers and the bias measurements belong to the same argument.
Evaluating Large Language Models Trained on Code — Chen, M., et al. (2021)⁴. Section 2 is the clearest published explanation of why the naive pass@k estimator is biased and what to use instead. The rest is the origin of HumanEval.
Holistic Evaluation of Language Models (HELM) — Liang, P., et al., Stanford CRFM¹². A live leaderboard built on the argument this page opens with: one number is never enough, so report accuracy, calibration, robustness, bias, toxicity and efficiency side by side.

References

The works cited through the text, in order of appearance:

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners — OpenAI technical report. Reports results in bits per byte precisely so that models with different tokenizers can be compared; section 3 explains the normalization. ↩
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020). BERTScore: Evaluating Text Generation with BERT — ICLR. ↩
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2021). Measuring Massive Multitask Language Understanding — ICLR. The MMLU benchmark, 57 subjects. ↩
Chen, M., Tworek, J., Jun, H., et al. (2021). Evaluating Large Language Models Trained on Code. Introduces HumanEval and the unbiased pass@k estimator; reports Codex at 28.8% (pass@1) and 70.2% (pass@100). ↩↩↩↩↩
Cobbe, K., Kosaraju, V., Bavarian, M., et al. (2021). Training Verifiers to Solve Math Word Problems — the GSM8K dataset. ↩
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019). HellaSwag: Can a Machine Really Finish Your Sentence? — ACL. ↩
Lin, S., Hilton, J., & Evans, O. (2022). TruthfulQA: Measuring How Models Mimic Human Falsehoods — ACL. ↩
Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — NeurIPS Datasets and Benchmarks. Reports over 80% agreement between a GPT-4 judge and human preferences — "the same level of agreement between humans" — and measures position, verbosity and self-enhancement biases. ↩↩↩↩
Chiang, W.-L., Zheng, L., Sheng, Y., et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference — ICML. ↩
Sainz, O., Campos, J. A., García-Ferrero, I., Etxaniz, J., de Lacalle, O. L., & Agirre, E. (2023). NLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for each Benchmark — Findings of EMNLP. ↩
Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2024). RAGAS: Automated Evaluation of Retrieval Augmented Generation — EACL (demo). Source of the four-metric decomposition above. ↩
Liang, P., Bommasani, R., Lee, T., et al. (2023). Holistic Evaluation of Language Models — TMLR. ↩