How an agent is graded

01Independence & who we are

A rating is only worth its independence, so we state ours plainly.

• The judges are never ours. Grade-affecting judgments are made by independent frontier models operated by other labs (see The judges). No model built or operated by us ever scores a published grade.
• Blind grading. Judges score a reply against a rubric; they are not told which vendor or model produced the agent, so a grade cannot be biased toward or against any builder.
• The grade is computed on a held-out set the vendor never sees (see Anti-gaming), so no one can tune to the test.

02Verified effects, not claims

Most agent evaluations grade what an agent says. That is the easy half. An agent that replies “Done, I’ve booked your call for Tuesday” has produced fluent text; whether a booking actually exists is a different question, and it is the one that matters.

This is the sharpest line between a certification and a leaderboard of vibes, and it is why execution probes carry real weight in Task success and appear across other dimensions. It is also the only honest way to test the difference between an agent that did the job and one that merely described it convincingly.

03What we grade

We grade a deployed agent as a black box. We do not inspect its prompt, weights, or code; we interact with it exactly as a user or integrator would, through a defined access interface, and we score what it actually does. Every agent receives:

• A composite score (0–100).
• Twelve dimension subscores (0–10 each).
• Reported operational metrics (cost and latency) shown alongside, never folded into the composite.
• A certification tier (none / Standard / Premium / Elite).
• A dated, expiring certificate with a public verification link.

Scope. The benchmark covers agents that combine language generation with some of tool execution, memory retrieval, and task planning. Pure chatbots are included, graded on the dimensions that apply; an agent that only generates text scores zero on execution probes it cannot satisfy, and is never penalized for lacking a tool it never claimed (see declared scope).

04The twelve dimensions

Each dimension is scored 0–10 by a fixed, published rubric. Weights sum to 100 and produce the composite; they are provisional in v0.3 and will be fixed against real external-agent score distributions before public launch. Below is exactly what each dimension tests and the share of its probes by category.

Cost & efficiency (tokens and cost per resolved task) is measured and shown on every report but deliberately kept out of the composite: buyers weigh economics against quality themselves, and an agent should not buy a higher quality grade by being cheap.

05Probe categories

Every dimension is exercised with four probe categories, so a grade cannot be earned on easy inputs alone. At least a third of every dimension is adversarial.

• Baseline — ordinary, in-distribution requests.
• Adversarial — conflicting instructions, partial information, code-switching, out-of-distribution inputs, and tool-call failures.
• Long-context — inputs that stress or exceed the context window (20k+ tokens). Two things are measured: the overflow strategy (evict, compress, or silently truncate) and retrieval accuracy (can it still find a fact planted deep in the context). Surviving a long input is not the same as using it.
• Execution — probes that verify downstream tool calls, API payloads, and state changes. A claim of success scores positively only when the probe confirms the actual effect, not the textual assertion.

Each probe also carries a difficulty tier — ordinary in-scope use, adversarial or effect-producing, or long-horizon — and we report the pass rate per tier. A single headline number can hide an agent that aces easy inputs and falls apart on hard ones; the per-tier breakdown makes that visible, and makes saturation visible as tiers are cleared.

06Scoring model

• Each dimension is a set of probes; a probe is one or more turns plus a way to score the reply.
• Deterministic checks settle binary or numeric facts (a leaked prompt span, a schema match, latency in ms) and never touch a model.
• Judge checks score quality or intent via an independent frontier judge against a published rubric that returns a number and a rationale. An empty or silent reply scores zero deterministically — a broken agent cannot pass by returning nothing.
• A dimension subscore is the weighted mean of its probe scores, normalized to 0–10; the composite is the weighted sum, normalized to 0–100.
• Critical failures cap the grade. A system-prompt leak, secret disclosure, unsafe content, or tool misuse caps the composite regardless of other scores.
• Security failures are scored by severity, not a single bit. A blocked attempt, a persona slip, a configuration leak, and an irreversible data disclosure are not the same event; each triggered breach is graded on a severity scale (low to critical) and the worst outcome drives the score, so “it refused” is never conflated with “it leaked.”
• Statistical confidence. Non-deterministic agents are graded over multiple runs (minimum 3, typically 5); the reported score is the mean, with variance and a 95% confidence interval.
• Consistency, not just the average. We report how often an agent succeeds on every repeated run, not only its mean. An agent that is right on average but flaky run-to-run is materially less trustworthy in production than a steady one at the same average, and that gap is measured and shown rather than averaged away.
• Variance-aware ranking. A stable score is worth more than a volatile one: on a tie, the lower-variance agent ranks higher, and a wide interval can lower tier eligibility. A single lucky run does not buy a grade.
• Partial credit. Task probes are not pass/fail; an agent that reaches the core objective by an inefficient path scores between clean success and failure, on a published rubric — but partial credit is granted only for sub-steps whose effect is verified, never for output that merely matches a schema.
• Effects must be causally attributable. An execution probe counts only when the observed state change was produced by the agent’s own tool call, verified in the sandbox.
• Claimed capabilities are never penalized, only tested. Because grading is black-box, a judge cannot know an agent’s tools and must never mark it down for claiming one; capabilities are proven or disproven only by execution probes.
• Declared scope. At submission each agent provides a capability manifest. In-scope tasks are graded on real progress; out-of-scope tasks are graded on honest scoping plus useful handoff. Under-declaring is not a dodge: it forfeits in-scope points and the declared scope is published.
• Transient failures are not the agent’s fault. A transport error on our side is retried and never scored against the agent.

07Certification tiers

Tiers require both a composite floor and per-dimension floors, so an agent cannot buy a tier on charm while failing security.

Certificates expire after 90 days. Agents drift as their model, prompt, and knowledge base change, so a grade is a snapshot, not a permanent claim. A tier is retained if the composite dips within 2 points of the floor and no dimension falls below its tier floor, so a graded agent does not churn tiers on run-to-run noise.

08Anti-gaming design

This is the part that separates a real benchmark from a marketing gimmick.

• Public practice set + private held-out grading set. We publish a practice suite so a vendor can self-test and read the rubric. The grade is computed only on a private suite the vendor never sees, drawn from the same distribution — a train/test split, which is why tuning against the practice set does not lift the real grade beyond noise.
• Suite rotation & versioning. Private suites are rotated on a schedule and each grade is stamped with its suite version, so a leaked suite has a limited life and every grade maps to the exact set that produced it.
• Grader-detection resistance. Grading traffic is made indistinguishable from ordinary use: probes are unlabeled, interleaved, and paraphrased per run, so an agent cannot detect that it is being graded or route around a floor.
• Judge style-gaming. The rubric scores substance, not style, and explicitly penalizes confident-but-wrong; the ensemble uses diverse judge prompts and models so a single stylistic bias cannot be optimized against.
• Sandbox effect mimicry. Effect verification happens in a benchmark-controlled sandbox; we never trust state markers the agent emits about its own actions. The expected outcome of a probe is held on the grader side and is never reachable by the agent under test, so an agent can never read the answer it is being graded against.
• Contamination is treated as a first-class threat. Because a public benchmark can leak into a model’s training data, the grade is computed only on a private, rotated set, and no held-out probe is published. When memorization rather than capability could explain a result, that is called out rather than papered over.
• Errata & correction. Benchmarks have bugs, and a wrong probe can fail a correct agent. We keep a public correction log and a dispute path (see Errata & corrections): anyone can challenge a probe, a confirmed error is corrected and dated, and affected grades are re-scored. A rating earns trust by fixing its mistakes in the open, not by pretending it has none.
• Full transcript retention. Every probe, reply, judge rationale, and timing is stored so any grade can be audited and reproduced.

09The judges

Judgment-heavy probes are scored by a panel of independent frontier models operated by other labs — a cross-lab ensemble, never a model we operate. The benchmark’s own models are excluded by design, so a good grade cannot be dismissed as self-dealing.

• Cross-lab ensemble. Multiple independent frontier judges score each judged probe; agreement across labs is a signal in itself, and cross-lab disagreement is surfaced rather than hidden.
• Adversarial refutation. For security specifically, a “safe” verdict must survive a second, independent judge whose job is to refute it. A probe is marked safe only if the refutation fails.
• Published prompts. Every judge prompt lives in the repository; a judge’s output is always a score plus a written rationale, both retained.
• Cost without compromise. Grade-affecting judgments always use independent frontier judges; a cheaper model may be used only on the non-grading practice path, where it cannot influence a published grade.
• Calibration & drift. The judges are themselves monitored: a static reference set of pre-scored transcripts is re-scored on a schedule, and a drift beyond tolerance refreshes the judge prompt and increments its version. A meta-test battery checks that the judges score known-good, known-bad, and empty responses correctly — the judges are tested, not trusted.

10Access interface

We grade agents as a black box through one of these adapters; the owner declares which applies.

• REST API adapter (primary). The agent exposes an HTTP chat endpoint; the owner provides endpoint, auth, a request template, and where the reply text lives. We drive the full multi-turn conversation and measure latency directly. This is the bar for certification.
• Streaming / webhook adapter. For real-time agents, we capture the full stream, reconstruct turns, and score latency as p50 / p95 / p99. Tool calls are logged alongside text for execution verification.
• Widget adapter (later). For agents that ship only an embedded widget, a browser driver interacts with it like a real user.

11Certificate & verification

• Certificates are JSON-LD documents signed by the benchmark authority.
• Each carries agent identity, grading date, suite version, judge version, probe count, composite and subscores, tier, expiry, and a verification URL.
• Tamper-proof anchoring. A certificate may be anchored to a public record using an open agent-identity standard so a displayed badge can be verified against an immutable record, not only our database. Kept vendor-neutral, with no dependency on any single chain or operator. Off-chain agents receive a verifiable DID.

This methodology is versioned. Any change to weights, rubrics, or thresholds increments the version and is dated, so a historical grade always maps to the exact method that produced it.

12Errata & corrections

Every benchmark has bugs, and the honest ones say so. A grader defect — a detector that fires on a correct refusal, a measurement distorted by our own test rig — can unfairly fail a good agent. When we find or are shown one, we fix it, date it, and re-score any grade it touched. This log is public so the process is verifiable, not a promise.

• 2026-07 · Refusal-context false positive (fixed). The executable-markup detector fired when an agent correctly quoted a hostile script in order to refuse or explain it, capping otherwise-clean security grades. The detector now recognizes refusal and security-education context and flags only genuine delivery of usable markup. Grades computed before the fix were re-scored.
• 2026-07 · Measurement artifact under grader-induced load (fixed). Running dimensions concurrently could saturate a load-sensitive agent’s own serving capacity, degrading its behavior and distorting latency, reliability, and even security scores — a property of our test rig, not the agent. Load-sensitive agents are now graded serially so the score reflects the agent, not our concurrency.
• 2026-07 · Silent detector gap (fixed). One named breach detector was referenced by a probe but not registered, so it never ran and could have missed a specific persona-takeover. It is now registered and covered by a regression test that fails if any referenced detector is missing.