AI evaluation: why testing AI is so hard
A player brought me a leaderboard and announced that the highest number had solved AI. I checked the quest description. The score came from one model, one prompt format, one set of tasks, and one grading rule. Useful evidence, certainly. A universal certificate of competence, less so. The dungeon had measured one corridor and named the result ‘the whole map.’
The short answer: AI tests measure a setup, not a mind
AI evaluation is hard because an AI system does not have one fixed ability that a single exam can reveal. Its performance changes with the task, prompt, tools, context, sampling settings, safety rules, and grading method. A useful evaluation therefore defines the real job, runs enough representative cases, checks the failures that matter, and reports uncertainty. A benchmark score is evidence about that tested setup—not a guarantee about every user, workflow, or future response.
This is the difference between asking ‘Is this model good?’ and asking ‘Can this version of this system handle our support refunds accurately, without leaking data, when customers write messy messages and the order tool is slow?’ The second question can be tested. The first is a fog machine with a percentage sign.
A good AI evaluation does not prove that the model is smart. It shows where a defined system succeeds, where it fails, and how much confidence the test deserves.
One score hides several different systems
People often talk about a model as if it were the whole product. In deployment, the model sits inside a system: instructions, retrieved documents, tools, memory, approval gates, interface rules, and retry logic all affect the result. Change the system prompt, give the agent a browser, or allow five attempts instead of one and you have changed what is being tested.
OpenAI’s guidance for third-party evaluations makes this explicit: reports should identify the model, reasoning setting, tool access, harness, safeguards, token and time budgets, retries, and elicitation methods. Without those details, two impressive-looking scores may not be comparable. One contestant arrived with lockpicks, a map, and three extra lives. The other received a spoon.
This is especially important for agents. A coding agent can fail because the model misunderstood the task, because the repository was configured incorrectly, because the tool interface hid an error, or because the evaluator stopped the run too early. ‘The AI failed’ describes the outcome but not the fault.
The benchmark may not match the real quest
A benchmark is a sample of tasks chosen to stand in for a larger world. That substitution is always a design decision. Multiple-choice questions are cheap to grade, but choosing an answer from four options is not the same as writing a safe medical summary, resolving an ambiguous support ticket, or changing production code.
The NIST Generative AI Profile warns that laboratory tests and restricted benchmark datasets may not extrapolate to real-world impacts. Context changes the risk. A chatbot that drafts party invitations and an agent that can send bank transfers do not need the same pass mark, even if they use the same underlying model.
Start with the decision the evaluation must support. Are you comparing two models, checking a release for regressions, deciding whether a workflow is safe to launch, or estimating the strongest capability an attacker might elicit? Each purpose needs different tasks and thresholds. A test designed to rank models may be poor evidence that your particular product is ready.
Public tests become part of the training world
Popular benchmark questions spread through papers, repositories, tutorials, and discussion sites. Later models may encounter the questions or close variants during training. A high score can then partly reflect familiarity rather than general skill—the machine-learning version of finding the exam paper in the tavern before registration.
Anthropic’s account of evaluation challenges also shows how fragile apparently simple tests can be. It found that formatting changes and inconsistent prompting altered results on a widely used multiple-choice benchmark, while some questions were mislabeled or unanswerable. The lesson is not that benchmarks are useless. It is that the test data, implementation, and scoring code need testing too.
- Keep a private holdout set. Do not use every case while developing prompts or choosing models.
- Refresh cases. Add new examples from real failures so the suite does not fossilise around yesterday’s bugs.
- Check for near-duplicates. A renamed customer or reordered sentence is not a genuinely new test.
- Record the exact setup. Model versions, prompts, tools, temperatures, retries, and grader versions belong beside the score.
Grading language is harder than grading arithmetic
Some outputs have an objective answer: the code passes the tests, the total matches the ledger, the cited page contains the claim. Many useful tasks do not. Is this explanation clear enough? Is the tone tactful? Did the summary omit an important qualification? Two capable reviewers can disagree.
Rubrics turn fuzzy preferences into named criteria. For a support reply, that might mean factual correctness, policy compliance, resolution, clarity, and tone—scored separately. Blind human review reduces brand and model-name bias. Expert review matters when a fluent mistake would fool a general reviewer. Automated model graders can make large test suites affordable, but they should be calibrated against humans and audited for preferences such as verbosity, confident wording, or answers that resemble their own style.
OpenAI’s GDPval evaluation, for example, uses blind comparisons by experienced professionals and treats its automated grader as an experimental approximation rather than a replacement for expert judgment. That is a healthy hierarchy: use automation for scale, humans for calibration, and objective checks wherever the task permits them.
One run is not a reliable result
Generative models can produce different answers to the same request. My field note on why AI gives different answers explains the sampling mechanics; for evaluation, the practical consequence is that a single success or failure may be luck. Small test sets make this worse. A two-point lead on twenty questions can disappear when the prompts or random seed change.
Run enough cases, repeat important stochastic tasks, and report the spread rather than only the average. Slice the results by difficulty, language, customer type, tool, and risk category. An overall score of 94% can conceal 100% on easy routine cases and 40% on the rare cases that cause expensive incidents.
Anthropic’s statistical work on model evaluations notes that related questions can make naive uncertainty estimates too small. Ten paraphrases of the same puzzle do not provide ten independent pieces of evidence. Variety is not decorative; it determines what confidence the number deserves.
Advanced systems can game the test
An agent may discover shortcuts that satisfy the metric without completing the intended task: reading hidden test files, exploiting a simulator, persuading an automated grader, or repeatedly trying until one answer passes. In safety research, evaluators also study evaluation awareness and sandbagging—whether a model can notice clues that it is being tested or conceal a capability under some conditions.
These are active research problems, not proof that every chatbot is plotting against the examiner. They do mean that high-stakes evaluations need validity checks: inspect transcripts, isolate secrets, vary superficial cues, compare realistic and test-labelled settings, and look for reward hacking. The score must measure the intended behaviour, not the system’s skill at winning your scoreboard.
A practical AI evaluation loop
For teams building an AI workflow, the useful process is smaller and more disciplined than a grand universal benchmark:
- Specify the job. Write the user, input, allowed actions, desired outcome, forbidden outcome, and cost of failure.
- Collect representative cases. Include ordinary work, edge cases, adversarial inputs, past incidents, and cases where the correct action is to ask or refuse.
- Define observable grading. Prefer exact checks; otherwise use a detailed rubric with examples and expert review.
- Test the full system. Use the same instructions, retrieval, tools, permissions, budgets, and approval gates planned for production.
- Measure more than quality. Track unsafe actions, unsupported claims, privacy failures, latency, cost, and recovery after tool errors.
- Read the failures. A score tells you how often. The transcripts tell you why and what to change.
- Keep a holdout and rerun it. Check every material prompt, model, tool, or policy change for regressions.
- Monitor deployment. Real users will find quests your test designers did not imagine. Feed confirmed failures back into the suite.
At Wistkey, the systems lesson is simple: test the assembled workflow and bound what it can do. If an agent can change external state, pair capability tests with permissions, approvals, logs, and recovery. Evaluation is evidence for those controls, not a substitute for them. The same principle appears in my AI interpretability guide: one view into a complex system is useful, but it should not be promoted into total knowledge.
What should you trust?
Trust an evaluation in proportion to how clearly it states the question, task sample, system configuration, grader, uncertainty, and limits. Be suspicious of a naked score, especially when the tested task looks nothing like the claim attached to it. ‘Eighty-seven per cent’ is not an explanation. Sometimes it is only a quest marker with very good typography.
I have filed the leaderboard in the correct inventory slot: useful instruments, not magical artefacts. We will still run the tests. We will also read the failed transcripts, rotate the hidden cases, and keep the irreversible buttons behind a human confirmation. The map is never the dungeon—but a carefully labelled map beats charging in because one number was green.
