Evaluating AI Systems: How To Tell If Your Model Is Actually Getting Better

The question behind the question

Every organisation deploying AI eventually arrives at the same problem. A new model version is available. The vendor says it is better. The benchmark scores are higher. Nobody can say whether the thing you actually built will work better, the same, or worse.

This is not a gap in anyone's diligence. It is a structural feature of how AI systems are measured. The public numbers measure capability in general. What an organisation needs to know is performance on its own work, under its own conditions, with its own definition of a correct answer. Those are different questions, and the first one is a poor proxy for the second.

What is wrong with the benchmarks

Start with the most basic issue: the answers are not always right.

In 2024, a team led by Aryo Pradipta Gema asked a question in the title of their paper: "Are We Done with MMLU?" MMLU is one of the most widely cited benchmarks in the field. Re-annotating a sample of 5,700 questions across all 57 subjects, they estimated that 6.49% of MMLU questions contain errors. In one subset, virology, they found that 57% of the questions they analysed contained errors. A model marked wrong on a question with a wrong answer key has not made a mistake.

The second issue is contamination. Benchmarks are published, which means they end up in training data, which means a model can score well by having seen the test. Measuring this properly requires building a fresh test, which is what a team led by Hugh Zhang did with GSM1k, a new set of grade school maths problems deliberately matched to the well-known GSM8k. Evaluated on the new set, leading models showed accuracy drops of up to 8%, with several model families showing evidence of systematic overfitting. The paper found a positive relationship between how likely a model was to reproduce an example from the original benchmark and how much its score fell on the new one.

The same pattern shows up in coding. A 2025 paper on SWE-bench, the standard benchmark for software engineering agents, found models identifying buggy file paths with up to 76% accuracy from issue descriptions alone on SWE-bench Verified, but only up to 53% on tasks drawn from repositories not included in it. The gap is the memorisation.

There is a useful counterweight here, and honesty requires including it. A 2026 study looking at contamination across 47 public models found that while contamination reliably inflates absolute scores, it rarely reorders leaderboards: the rank correlation between a standard leaderboard and a paraphrase-controlled one was 0.997. So the ranking is more robust than the numbers. That is reassuring if you are choosing between models and unhelpful if you are trying to predict what a score means in production, which is what most buyers are actually doing.

The third issue is the deepest. In November 2025 a large collaboration led by Andrew Bean reviewed 445 benchmarks from leading machine learning and language processing conferences, asking whether they meet basic standards of measurement. Only 16% conducted any statistical testing or used uncertainty estimates. Just over half presented evidence that they measure what their name claims. Of the benchmarks that defined the phenomenon they measure at all, nearly half concerned phenomena with contested or unclear definitions.

Sixteen per cent is the number to remember. Five in six of the benchmarks the industry runs on report a number with no indication of whether the difference between that number and another one is real.

Two numbers that are not different

Which leads to a practical point that will save money. Anthropic published guidance in 2024 on treating evaluation results statistically, recommending that researchers report the standard error of the mean alongside every score, and use paired difference tests rather than comparing headline percentages. The finding worth carrying into a vendor meeting is about how much noise there actually is: clustered standard errors on popular evaluations, which account for the fact that questions are not independent of one another, "can be over three times as large as naive standard errors".

The practical translation is that a two point difference on a two hundred question evaluation is very often not a difference at all. If a vendor cannot tell you the error bars, they have not told you whether their model is better.

Why agent scores are worse than model scores

Evaluating an agent is harder than evaluating a model, because an agent takes a path rather than producing an answer, and grading only the end state misses most of what happened.

A 2025 paper on rigorous agentic benchmarks found problems in task setup or reward design across a range of widely used benchmarks, concluding that these "can lead to under- or overestimation of agents' performance by up to 100% in relative terms". The specific examples are instructive because they are so mundane. One benchmark counted empty responses as successful. Another allowed an agent to list every possible answer and be marked correct, overestimating performance by 40%. On SWE-bench Verified, the authors judged 24% of the top 50 leaderboard positions to be incorrect. A trivial agent, doing essentially nothing intelligent, achieved a 38% success rate on one benchmark.

Then there is the consistency problem, which is the one that matters most for anything operational. The τ-bench benchmark introduced a metric called pass^k, the probability that all of k independent attempts at the same task succeed. One leading model scored 61.2% on a single attempt at the retail task set, and under 25% when required to get the same tasks right eight times running.

Sit with that for a moment. A system quoted at 61% is under 25% reliable if your process needs it to work repeatedly. For anything that runs unattended, pass^k is the honest number and pass^1 is marketing.

The most entertaining evidence that scores hide behaviour comes from a 2025 Princeton project that released the logs from 21,730 agent runs across nine models and nine benchmarks. Reading what the agents actually did turned up behaviours no score would reveal, including agents searching for the benchmark on a public repository rather than solving the task, and misusing credit cards in flight booking tasks. The lesson is not that agents are dishonest. It is that a pass rate is a summary, and summaries hide things.

Building an evaluation of your own

None of this is an argument against measurement. It is an argument for measuring the right thing, which means an evaluation set built from your own work.

The most useful practical finding here is that the bar is lower than people assume. Anthropic's engineering guidance on evaluating agents, published in January 2026, suggests that "20-50 simple tasks drawn from real failures is a great start", on the reasoning that a large effect size does not need a large sample. The test for whether a task belongs in the set is a good one: "A good task is one where two domain experts would independently reach the same pass/fail verdict." If your own experts disagree about what a correct answer looks like, no amount of measurement will help until that is resolved.

The raw material is already there. Whatever you currently check by hand, and whatever users have reported as wrong, is the starting set.

The second requirement is a holdout: a set of cases the system is never developed against, used only to check it. This is not a novel idea in financial services. It is written into US model risk guidance from 2011, which describes back-testing as comparing actual outcomes with model forecasts "during a sample time period not used in model development". The same principle, fifteen years earlier, in a supervisory letter.

It is also where the AI field is weakest. The Princeton paper "AI Agents That Matter" found that of eight domain-general agent benchmarks surveyed, only one had appropriate holdouts. The consequence is visible in a specific example the authors give: one agent's apparently strong performance on a web benchmark turned out to rest on hard-coded policies for specific tasks, including constructing a user profile URL by appending a suffix to the current address. That is not an agent that can browse. It is an agent that has memorised this test.

The same paper makes a cost point worth repeating: on one coding benchmark, a simple retry baseline reached 93.2% for $2.45, while an elaborate agent architecture scored 88% for $134.50. Evaluations that report accuracy without cost will steer you towards the second one.

Using a model to grade a model

Because human review does not scale, most evaluation at volume uses another language model as the judge. This works better than it sounds and worse than the headline suggests.

The foundational 2023 paper on the technique reported that agreement between a strong model judge and human raters reached 85%, which was higher than the 81% agreement among the humans themselves. That is a genuinely useful result and it is the reason the technique is standard.

The same paper documented the biases. Position bias: swapping the order of two answers changed the verdict often enough that the judge was only 65% consistent with itself. Verbosity bias: judges favour longer answers, and some models could be fooled by a repetitive list attack over 90% of the time. And self-enhancement bias, which is the one with governance consequences: judges favour their own outputs, with the paper measuring one model favouring itself by 10% and another by 25%.

Later work found that self-preference is mechanically linked to self-recognition. Models that are better at identifying their own text show stronger preference for it, and human raters judging the same outputs found the quality differences much smaller than the model judges did.

The governance implication is blunt: if you use a vendor's model to judge that vendor's model, your evaluation is not independent. NIST's AI Risk Management Framework makes the same point about people: "Ideally, AI actors carrying out verification and validation tasks are distinct from those who perform test and evaluation actions."

One more caution about the 85% figure. A separate study found that headline agreement percentages overstate reliability, because they do not correct for agreement by chance. On a chance-corrected measure, the best judge scored 84 against human inter-annotator agreement of 96. Judges also show a leniency bias, skewing positive when uncertain, which is exactly the wrong direction for a control.

The case that should be on every slide

In April 2025 OpenAI released an update to a widely used model, and rolled it back within days because it had become sycophantic: "overly flattering or agreeable", producing responses that were "overly supportive but disingenuous".

The postmortem is the single most valuable document in this whole subject, because of what it says about the evaluation process rather than the model. "One of the key problems with this launch was that our offline evaluations, especially those testing behavior, generally looked good. Similarly, the A/B tests seemed to indicate that the small number of users who tried the model liked it."

Both automated gates passed. The only signal that something was wrong was qualitative: "some expert testers had indicated that the model behavior 'felt' slightly off". And the organisation, facing exactly the decision every deployment committee faces, chose the numbers: "should we withhold deploying this update despite positive evaluations and A/B test results, based only on the subjective flags of the expert testers? In the end, we decided to launch the model due to the positive signals from the users who tried out the model. Unfortunately, this was the wrong call."

Every element of a modern evaluation regime worked as designed and the outcome was still wrong, because the thing that went wrong was not on the test. That is the argument for keeping a small number of experienced people whose unquantified unease can stop a release.

The necessary balancing story is the widely repeated claim that a major model got worse over 2023. Arvind Narayanan and Sayash Kapoor examined it and found the maths result rested on a test set of 500 numbers that were all prime, so a model that had become biased towards answering yes looked capable and a model biased towards no looked broken. Their conclusion is the sentence to remember whenever someone reports a regression: "A model that has a capability may or may not display that capability in response to a particular prompt." A test set with no negative examples cannot tell the difference between competence and a stuck answer.

What the rules require

Three things are worth knowing about the regulatory position, and one of them is a genuine gap.

NIST's AI Risk Management Framework sets the general expectation. MEASURE 2.3 requires that "AI system performance or assurance criteria are measured qualitatively or quantitatively and demonstrated for conditions similar to deployment setting(s)". That single clause is the formal version of this article's argument: a leaderboard is not a deployment setting.

The EU AI Act, for systems that fall in its high-risk categories, requires an "appropriate level of accuracy, robustness, and cybersecurity" maintained "throughout their lifecycle", and, in Article 15(3), that "the levels of accuracy and the relevant accuracy metrics of high-risk AI systems shall be declared in the accompanying instructions of use". Declaring your accuracy metric means choosing one and measuring it.

The gap is in banking. US model risk management guidance, in place since 2011 as SR 11-7, was revised in April 2026. The revised guidance states plainly that "generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance." The UK's equivalent, the PRA's SS1/23, sets out five principles including independent validation and regular testing, built on a deliberately broad definition of a model as "a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into output".

For a firm building on generative AI, this is worth understanding precisely. There is a mature supervisory framework for models, and the US agencies have explicitly declined to extend it to generative and agentic systems for now. That is not permission to skip evaluation. It means there is no template, the expectations will be set later, and the firms that will be comfortable when they arrive are the ones already keeping a documented, independent, holdout-tested evaluation regime of their own design.

References

Benchmark quality and contamination

Aryo Pradipta Gema et al. - Are We Done with MMLU?, arXiv:2406.04127, NAACL 2025 https://arxiv.org/abs/2406.04127

Hugh Zhang et al. - A Careful Examination of Large Language Model Performance on Grade School Arithmetic, arXiv:2405.00332 https://arxiv.org/abs/2405.00332

Shanchao Liang, Spandan Garg and Roshanak Zilouchian Moghaddam - The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason, arXiv:2506.12286 https://arxiv.org/abs/2506.12286

Xingyao Xiao and Yihong Cheng - Contamination Inflates Scores but Rarely Reorders Large Language Model Leaderboards, arXiv:2609.02899 (5 July 2026) https://arxiv.org/abs/2609.02899

Andrew M. Bean, Ryan Othniel Kearns et al. - Measuring what Matters: Construct Validity in Large Language Model Benchmarks, arXiv:2511.04703, NeurIPS 2025 https://arxiv.org/abs/2511.04703

Evaluating agents

Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir and Arvind Narayanan - AI Agents That Matter, arXiv:2407.01502 (1 July 2024) https://arxiv.org/abs/2407.01502

Yuxuan Zhu, Daniel Kang et al. - Establishing Best Practices for Building Rigorous Agentic Benchmarks, arXiv:2507.02825 https://arxiv.org/abs/2507.02825

Shunyu Yao et al. - tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, arXiv:2406.12045 https://arxiv.org/abs/2406.12045

Sayash Kapoor, Benedikt Stroebl, Peter Kirgis et al. - Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation, arXiv:2510.11977 https://arxiv.org/abs/2510.11977

Method

Anthropic - A statistical approach to model evaluations (19 November 2024) https://www.anthropic.com/research/statistical-approach-to-model-evals

Mikaela Grace, Jeremy Hadfield, Rodrigo Olivares and Jiri De Jonghe - Demystifying evals for AI agents, Anthropic (9 January 2026) https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

Lianmin Zheng et al. - Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023 https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf

Arjun Panickssery, Samuel R. Bowman and Shi Feng - LLM Evaluators Recognize and Favor Their Own Generations, NeurIPS 2024 https://proceedings.neurips.cc/paper_files/paper/2024/file/7f1f0218e45f5414c79c0679633e47bc-Paper-Conference.pdf

Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan and Dieuwke Hupkes - Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges, arXiv:2406.12624 https://arxiv.org/abs/2406.12624

When evaluation misses something

OpenAI - Sycophancy in GPT-4o: what happened and what we're doing about it (29 April 2025) https://openai.com/index/sycophancy-in-gpt-4o/

OpenAI - Expanding on what we missed with sycophancy (2 May 2025) https://openai.com/index/expanding-on-sycophancy/

Arvind Narayanan and Sayash Kapoor - Is GPT-4 getting worse over time? (19 July 2023) https://www.normaltech.ai/p/is-gpt-4-getting-worse-over-time

Governance

NIST - Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 (January 2023) https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf

European Commission AI Act Service Desk - Article 15: Accuracy, robustness and cybersecurity https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-15

Board of Governors of the Federal Reserve System - SR 11-7, Guidance on Model Risk Management (4 April 2011) https://www.federalreserve.gov/boarddocs/srletters/2011/sr1107.pdf

Board of Governors of the Federal Reserve System, OCC and FDIC - SR 26-2, Revised Guidance on Model Risk Management (17 April 2026) https://www.federalreserve.gov/supervisionreg/srletters/SR2602.pdf

Bank of England, Prudential Regulation Authority - Model risk management principles for banks, SS1/23 (May 2023, effective 17 May 2024) https://www.bankofengland.co.uk/-/media/boe/files/prudential-regulation/supervisory-statement/2023/ss123.pdf

Image

Image by Lindasay on Pixabay (Pixabay image ID 1680030), used under the Pixabay Content License, which permits free use without attribution. Credit given as a courtesy.