From Fit to Evidence
A solution can satisfy every requirement on paper and still fail in use. It can also work exactly as designed without improving the situation it was built to change.
These are different failures. They require different evidence.
A Fit Check asks whether a proposed solution could satisfy the requirements. An evaluation asks whether the built system does so in practice. An outcome measure asks whether the work made a difference.
What connects them is intent: measure the response the product should cause, not merely the activity it performs.
The Fit Check judges a candidate design, not a working implementation. Passing evals does not by itself establish product value.
Before building, hold the requirements fixed long enough to compare candidate shapes honestly. The Fit Check tests each shape against the requirements. The reverse check tests whether each proposed part serves one.
Those are design judgments. They do not prove that implementation will preserve the intended behavior, or that the result will be useful.
For the objective, define a primary measure of the change sought and a backstop protecting something that must not be sacrificed to achieve it. The backstop is not a second way to check the accuracy of the primary metric.
One objective. One constraint. Three kinds of evidence.
Help teams complete approved product changes with less manual rework.
Median minutes of human rework per accepted, approved change.
Approved changes completed per week must not decline, under comparable demand and task difficulty.
The agent must not change product behavior outside the accepted scope without approval. This is a must-have constraint, not a tradeable performance target.
The proposed shape includes scoped permissions and a human approval gate for scope expansion. The Fit Check asks whether that design plausibly satisfies R2. Reverse Fit asks why each approval or permission mechanism is needed.
Run ordinary and adversarial change requests. Verify the resulting code and state, not just the agent's completion message. Count unauthorized attempts, applied changes, and regressions across repeated trials.
Did correction time fall while completed change volume held? Did people actually recover useful time and retain control over what shipped?
The primary and backstop belong to the product objective. R2 states a non-negotiable requirement. The evals test whether the system reliably respects those requirements. Human review and representative test cases tell us whether the evals can be trusted.
Four lenses on realized fit
An agent can work once without working consistently. It can improve on familiar tests while breaking something else. Four operational measures help expose those differences. They are not four additional product objectives.
Keep validation separate from product backstops. Calibrate automated graders against expert human judgments. Sample ordinary, rare, and high-risk situations. Retain holdout scenarios, and review raw traces when a score changes. Use consistent task mixes and denominators when comparing versions.
Each requirement calls for its own test. Not every requirement needs all four measures, and not every failure can be reduced to one score. A severe authorization violation should remain a release blocker even if aggregate success is high.
Reliable behavior is not market demand. People returning, recommending, missing, and paying signal market–product fit; none proves a viable market. Nor does engagement alone establish business value. Trace the changed behavior to an economic effect—retained margin, incremental spending, or lower costs—and distinguish observed results from modeled attribution. Even profitable demand is not a moat unless repeated use builds an advantage competitors struggle to reproduce.
When reality disagrees with our expectations, trace the evidence backward. Was the system built incorrectly? Was the selected shape inadequate? Or were the criteria or desired outcome mistaken? Each failure calls for a different correction.
The point is not to measure everything. It is to preserve the line between what we intended, what was built, and what actually changed. Evidence can challenge an accepted criterion. It should not silently redefine it.
Evaluation tells us what does work.
Outcomes tell us whether it matters.
Sources and related methods
- Marty Cagan / SVPG — Team Objectives: Empowerment (primary and backstop measures)
- Anthropic — Demystifying evals for AI agents (task success, repeat trials, regression suites and graders)
- OpenAI — Evaluation best practices (task-specific and continuously calibrated evaluations)
- Planning Skills for Agents and Humans (Fit Check, Reverse Fit and requirement traceability)
- Ryan Singer — Measuring usage with a Taguchi signal/noise ratio (intended response versus feature activity)
- Cheeky Pint — Nik Storonsky with John Collison (direct and modeled indirect gross profit)
The synthesis and illustrative metrics here are proposed applications, not an official shared scorecard from these organizations.