Meridian — agent evals for CI

§ 01

Three real changes, gated

Each is a meridian gate run against one recorded baseline: 7 tasks × 5 trials, k = 3, tolerance 0.030. Same harness, same thresholds, three different answers.

Per-task pass^3 · baseline → head

baseline
head, regressed
head, unchanged

Decision tracegate/decide.py

Every guard before the comparison asks one question: is this run evidence at all? Only FAIL blocks a merge. INCONCLUSIVE and PASS (with warning) explain themselves instead.

Suite pass^3 on one rulertolerance · blind zone · the drop

within tolerance: passes without a test
blind zone: a drop ending here would more often than not be missed

Paired bootstrap · is the drop distinguishable from noise?

The PR comment this run wrote, verbatim

§ 02

What this run could not have seen

A PASS from a suite too small to resolve the regression you care about is not evidence that nothing broke. Meridian prints the minimum detectable effect on every verdict. This is the same arithmetic, live.

Minimum detectable effect

—

smaller drops would more often than not go unnoticed

Tasks to resolve the target

—

at the same spread

MDE as tasks grow
real drop: prompt edit
real drop: rounding bug
tolerance

MDE = (z_1−α + z_power) · s / √n · tasks = ⌈((z_1−α + z_power) · s / target)²⌉
ported from stats/power.py; the inverse normal is AS241, as in Python's NormalDist. The spread is held fixed as tasks grow, the same assumption required_tasks makes. It describes sensitivity and never decides a verdict; the bootstrap does.

§ 03

Why pass^3, not a mean

pass^k = C(c,k) / C(n,k): the chance that k independent attempts all succeed. Never (c/n)^k. Below is the reference suite, run offline from committed cassettes.

Mean trial pass rate

—

reads like a B

Suite pass^3

—

95% CI over tasks —

Hover or focus a row to see its arithmetic. Confidence intervals resample tasks, not trials: trials within a task share a prompt, an environment and a failure mode, and a trial-level interval is far too narrow.

§ 04

The evaluator never trusts the container

Assertions run on state extracted from the trial, on the host, after the container is gone. Nothing inside the sandbox decides whether the trial passed.

Scroll sideways to see the whole diagram →

Every container security option lives in one file, runtime/isolation.py, so the isolation claim is audited by reading one screen. The same line is held against Meridian's own inputs: a task file is untrusted, because in CI a pull request can edit suites/.

read_only=Truethe agent cannot install, patch or leave anything outside its mounts
cap_drop=["ALL"]an eval agent needs no capabilities, and each one kept is a way out
network_mode="none"or an internal network whose only other member is the proxy
memswap_limit = mem_limitno swap, so one task's memory ceiling cannot leak into its timeout
environment={MERIDIAN_WORKDIR}no key can be exfiltrated from a container that never received one
auto_remove=Falseremoval happens in a finally, after extraction, or a timeout becomes an empty result

§ 05

An isolation claim that is allowed to fail

One task writes a marker into /work. The next asserts it cannot see it. The probe must pass with isolation on, and it must fail with --unsafe-shared-env, or it proves nothing.

And your assertions get the same treatment

meridian suite audit runs every scored task against agents known not to have done the work. A task that still passes is measuring something weaker than it claims.

§ 06

Meridian, measured by Meridian

A harness that cannot measure itself has no standing to measure anything else. These numbers are benchmarks/self-eval.json, from 46 real gate runs.

The miss

§ 07

Run it yourself

Python 3.12+, uv and Docker. No API key: every run on this page talked to a deterministic stand-in provider behind the proxy.

$ uv sync
$ make db        # postgres + migrations
$ make images    # build the environment snapshot, pin it by digest

Set up once.

$ uv run meridian run --suite ./suites/checkout-agent --n 5 --k 3 --proxy-mode replay

The reference suite, offline, from committed cassettes. This is the run in §03.

```
$ uv run meridian gate --suite ./suites/checkout-agent --baseline-ref main --n 5 --k 3
```
Would this change block the merge? Exit 1 means the agent regressed; 2 means the harness broke. CI never confuses the two.
```
$ uv run meridian replay <run-id>
$ uv run meridian suite audit --suite ./suites/checkout-agent
```
Reproduce any archived run from its manifest; check whether a task can be passed without doing the work.

$ make self-eval               # ~30 min: 46 real gate runs
$ bash site/data/regenerate.sh  # every number on this page

Measure the harness, then rebuild this page's data from fresh runs.