Stop scoring your agent zero at 90% done.

Binary pass/fail is a correct hard gate and a terrible hill-climbing signal. Score the run continuously — then watch how much signal binary erases.

Built on Ken Huang's "Stop Scoring Agents Zero at 90% Done". Benchmark numbers below are from his piece (Android Bench 2.0).

See it: the signal binary erases

Real leaderboard numbers from the essay. Pick an agent-harness run — binary pass rate vs. continuous completion.

32.7%

Binary pass rate

84.7%

Continuous completion

Claude Opus 5.5 · claude-code — avg latency 15.2h · avg cost $215.4 · CI 18.7–47.3

📉 Binary says "failed." Continuous says 84.7% of the engineering work landed. Same run — one score erases the other.

"You cannot improve what your score collapses to a coin flip." — Ken Huang

Build it: your continuous rubric

Score a recent agent run across dimensions. Binary stays the headline — completion is the hill-climbing supplement.

Broken invariants (penalty multipliers)

Regression broken ×0.5

Security/policy violation ×0.3

Binary gate — did the run fully pass?

100%

Binary

–

Completion

Anti-patterns (from the essay)

Binary-only rubrics — erase 90% progress and teach your leaderboard to lie.
Brittle pixel-equality visual checks — noisy up to 47.5% differences in calibration. Android Bench replaced pixel equality with a multimodal visual judge scoring 0.0–1.0.
Waiting for a public leaderboard — "Do not wait for a public Android-style leaderboard in your vertical. Build the rubric first."

🔬 The exact-keys pattern (named, not specified)

The essay names two further patterns without full mechanics: exact-key simulated estates (grade against exact keys inside simulated worlds — answers whether the answer is right when production is a mess of CRMs and tickets) and typed-decision judges (gate and judge with typed decisions). They're design directions, not recipes — the continuous-rubric design above is the fully specified mechanic.

Share this rubric

Replay link — anyone who opens it sees your dimensions, weights, and the binary-vs-completion delta.

Retake with fresh inputs →