Benchmarks and LLM judges are both shaky — how to evaluate AI agents

Executive summary

Both tools used to decide “which model is better” are shaky at the same time. Public benchmarks get contaminated, and LLM judges, the usual alternative, are consistently wrong. One conclusion remains: do not trust other people's scores; measure on your own tasks.

The benchmark's creator abandoned it first

SWE-bench Verified was the de facto standard for measuring coding agent performance. Model launch after model launch reported this score. Then, in 2026, OpenAI said it would no longer evaluate its own models on this benchmark.

The trigger was that scores were rising too fast: from 74.9% to 80.9% in six months. When OpenAI looked into whether the remaining 20% were genuinely hard problems, it found something else.

2.1 Audit design

Of the 500 tasks in total, 138 (27.6%) were audited. This was not a random sample. OpenAI picked the tasks that OpenAI o3 consistently failed to solve across 64 independent runs. At least six experts reviewed each task independently, and the issues they flagged were checked again.

2.2 What turned up

Of the 138 audited tasks, 59.4% had serious flaws in the test design or the problem description. This is not only about tests: flaws in the problem descriptions themselves are included.

2.3 There was more than one cause

OpenAI presented two problems, separately. They should not be lumped together.

To check the second problem, OpenAI repeatedly prompted GPT-5.2-Chat, Claude Opus 4.5 and Gemini 3 Flash Preview, for up to 15 turns per task. Every frontier model tested (the latest, most capable models) either reproduced the gold patch (the reference solution) verbatim or output task-specific details verbatim. Gemini reproduced details of the gold patch when given nothing but the task ID.

As a replacement, OpenAI said it would use the public split of SWE-bench Pro, and that it is also building its own evaluations that are not released publicly. The reasoning is that anything public eventually gets contaminated.

Two ways contamination gets in

Benchmark contamination means the model has seen the test questions beforehand. There are two routes, and the more recent one, search-time contamination, is a problem that has newly emerged in the age of agents.

3.1 What search-time contamination (STC) is

A preprint released in June 2026 defined this as Search-Time Contamination (STC) and measured it. It describes an agent using search to bypass a problem it was supposed to solve by reasoning.

The measurements showed that search-time contamination was widespread and raised evaluation scores by up to 4 percentage points.

The paper's recommended countermeasures are isolated sandboxes, transparent search trajectories and controlled benchmark access. Evaluators should apply all three, and anyone interpreting a score should check whether each measure was applied during the evaluation.

LLM judges: consistent, but not correct

Because public benchmarks cannot be trusted, having an LLM do the grading has become standard practice. To measure things with no clean right answer, such as free-form answers or agent execution traces, there is often no other way.

A preprint released in June 2026 tested this approach at scale. It ran 21 LLM judges from 9 providers over 118 runs and analyzed about 541,000 individual verdicts.

4.1 Raw agreement overstates discriminative power

The kappa coefficient (Cohen's kappa) measures how much two sets of ratings actually agree after correcting for the chance that they agree by accident. The raw agreement rate commonly used to validate judges makes no such correction. On a question with two choices, that is like counting the roughly half you would get right by answering at random.

4.2 High reliability, low validity

The paper's title is Reliability without Validity. First, the two terms.

The paper's examples are two models. They illustrate this paradox; they do not describe a trend across all judges.

Here position bias is the value the paper defines as |P(A wins) − 0.5|. It shows how strongly the judge favors the answer placed first; 0 means the order has no effect. How often a verdict actually flips when the order is swapped is measured separately, by a different metric called the position flip rate. Do not mix up the two.

4.3 Rankings shift when the benchmark changes

The same judges' rankings moved a lot depending on the benchmark. However, the paper's own numbers disagree. The abstract says judge rankings shift “by up to 14 positions” across benchmarks, while the body says Llama 3.3 70B moved from 5th on MT-Bench to 20th on JudgeBench, a shift of 15 places. Either way the point is the same: which judge comes out ahead depends heavily on the benchmark you measured it on.

4.4 Verbosity bias was small

Verbosity bias (favoring longer answers), a common worry in practice, came out small in this study, at below 0.011. It was measured as the Pearson correlation between the difference in answer length and the verdict, and the result is limited to a single pairwise rubric on MT-Bench.

Where judges break down

RAND's Judge Reliability Harness is an open-source tool that deliberately stress-tests LLM judges. Give it a benchmark dataset and a judge configuration, and it generates reliability tests automatically. The paper was accepted at an ICLR 2026 workshop.

Four state-of-the-art judges were tested on four benchmarks covering safety, persuasion, misuse and agentic behavior, and not one judge was uniformly reliable across all the benchmarks.

The manipulations that broke them were nothing elaborate.

So what should you measure?

For practical decisions, the findings of the previous sections lead to the following. The right-hand column is the evidence; where that cell is empty, the row is my own judgment.

Building your own eval set

This is a minimal setup a small team can manage. It is not about building a complete evaluation system; it is about quickly getting something that beats nothing.

1. Collect 30–50 real cases. Do not make them up; use requests you actually handled. They have to be unpublished on the web to stay uncontaminated.
2. Write down “what counts as success” for each case. This is a pass criterion, not a score. Conditions nobody would argue about work best: was the file created, did the tests pass, is the amount correct.
3. Start by giving machines everything they can judge. Exit codes, regular expressions, schema validation and number matching belong here. This part needs no LLM judge, so there is nothing to wobble.
4. Use an LLM judge only for what is left. Then stress-test that judge: for example, feed it the same answer with only the formatting changed and check that the verdict stays the same.
5. Make the control run a fixture. Every time, also run a configuration in which nothing was changed. If its score moved, that day's comparison is void.

The context engineering report on this blog repeatedly suggested splitting runs into two conditions and comparing the results directly. The five steps above are the minimum procedure for re-measuring that way.

Checklist

What this is not yet

• The figures in this article are not our own measurements. They summarize three arXiv papers and a public OpenAI post.
• The three cited papers have different review status. Judge Reliability Harness is a paper accepted at an ICLR 2026 workshop; the other two (Reliability without Validity, Search-Time Contamination) are preprints that have not been peer reviewed. Do not give all three the same weight.
• Do not read 59.4% as an overall rate. The 138 audited tasks are a biased sample, selected toward tasks the model often got wrong. The rate is not out of all 500.
• SWE-bench's overall contamination rate has not been published. OpenAI presented individual cases only. If you see material that puts a number on the contamination rate, check its source.
• The search-time contamination study is limited to six medical QA benchmarks. Its main experimental subject is also a single system, Tongyi Deep Research. The paper gives no basis for carrying the figure of up to 4 percentage points over to coding or general reasoning benchmarks.
• The judge paper disagrees with itself on how far rankings moved. The abstract says 14 places; the body gives a 15-place case. This article reports both as they are.
• Test-retest reliability 0.95 / position bias 0.10 are the values of two models. They are not the trend across all 21 judges; the paper presents them as examples of the paradox.
• Position bias and verbosity bias use different scales. A deviation in probability and a correlation coefficient cannot be compared directly.
• I could not open the OpenAI original myself. Access was blocked, so I used quoted passages obtained through a tool that could read the original. Before you rely on it for an important decision, please check the original yourself.
• The five steps in section 7 and the checklist in section 8 are my practical judgment. The cited studies did not measure their effect.
• This field changes fast. Contaminated benchmarks get replaced, and the new benchmarks get contaminated in turn. It makes more sense to watch the structure than the individual numbers.

Sources

This document is a technical summary of material that was public as of 2026-09-07. Two of the cited papers are preprints, so check their latest versions and any replication results before using them for an important decision.