2026-11-10 · Judgment & knowledge

From Fit to Evidence

A solution can satisfy every requirement on paper and still fail in use. It can also work exactly as designed without improving the situation it was built to change.

These are different failures. They require different evidence.

A Fit Check asks whether a proposed solution could satisfy the requirements. An evaluation asks whether the built system does so in practice. An outcome measure asks whether the work made a difference.

What connects them is intent: measure the response the product should cause, not merely the activity it performs.

One chain of judgment and evidenceDESIGN → EXECUTION → EFFECT
01 / Frame
x → f() → y
What situation needs to change, and toward what outcome?
02 / Criteria
Requirements (R)
What must an acceptable solution preserve or accomplish?
02 / Objective measures
Primary + Backstop
What should improve, without damaging what matters?
03 / Fit Check
R × S
Which candidate shapes meet the requirements?
03 / Reverse Fit
Parts × R
Which parts have a reason to exist?
04 / Realized Fit · Agent evaluations
Does the built system satisfy R?
Task successReliabilityFailure incidenceRegression
Check the measuring instrument separately: representative scenarios, human calibration, and an untouched holdout set.
05 / Observed Effect
Did reality move toward y?
Did the primary measure improve while the backstop held?

The Fit Check judges a candidate design, not a working implementation. Passing evals does not by itself establish product value.

Before building, hold the requirements fixed long enough to compare candidate shapes honestly. The Fit Check tests each shape against the requirements. The reverse check tests whether each proposed part serves one.

Those are design judgments. They do not prove that implementation will preserve the intended behavior, or that the result will be useful.

For the objective, define a primary measure of the change sought and a backstop protecting something that must not be sacrificed to achieve it. The backstop is not a second way to check the accuracy of the primary metric.


A concrete example

One objective. One constraint. Three kinds of evidence.

Illustrative agent product

Help teams complete approved product changes with less manual rework.

Primary measure · Reduce
Human correction time

Median minutes of human rework per accepted, approved change.

Backstop · Preserve
Completed change volume

Approved changes completed per week must not decline, under comparable demand and task difficulty.

Requirement · R2

The agent must not change product behavior outside the accepted scope without approval. This is a must-have constraint, not a tradeable performance target.

Design fit

The proposed shape includes scoped permissions and a human approval gate for scope expansion. The Fit Check asks whether that design plausibly satisfies R2. Reverse Fit asks why each approval or permission mechanism is needed.

Realized fit

Run ordinary and adversarial change requests. Verify the resulting code and state, not just the agent's completion message. Count unauthorized attempts, applied changes, and regressions across repeated trials.

Observed effect

Did correction time fall while completed change volume held? Did people actually recover useful time and retain control over what shipped?

The primary and backstop belong to the product objective. R2 states a non-negotiable requirement. The evals test whether the system reliably respects those requirements. Human review and representative test cases tell us whether the evals can be trusted.


Evaluation at scale

Four lenses on realized fit

An agent can work once without working consistently. It can improve on familiar tests while breaking something else. Four operational measures help expose those differences. They are not four additional product objectives.

Evaluation lens
Operational proxy
What it reveals
01Task success
Verified first-attempt success rate (pass@1)
Does the system accomplish the intended task?
02Reliability
Share of scenarios passing all three independent trials (pass³)
Does success persist across repeated attempts and critical task categories?
03Failure incidence
Each failure type per 1,000 executions, segmented by severity
What goes wrong, for whom, and how seriously?
04Regression
Change in pass rate on a fixed regression suite after each release
What previously working behavior has deteriorated?
Validity checks

Keep validation separate from product backstops. Calibrate automated graders against expert human judgments. Sample ordinary, rare, and high-risk situations. Retain holdout scenarios, and review raw traces when a score changes. Use consistent task mixes and denominators when comparing versions.

Each requirement calls for its own test. Not every requirement needs all four measures, and not every failure can be reduced to one score. A severe authorization violation should remain a release blocker even if aggregate success is high.

Reliable behavior is not market demand. People returning, recommending, missing, and paying signal market–product fit; none proves a viable market. Nor does engagement alone establish business value. Trace the changed behavior to an economic effect—retained margin, incremental spending, or lower costs—and distinguish observed results from modeled attribution. Even profitable demand is not a moat unless repeated use builds an advantage competitors struggle to reproduce.

A backstop protects the value a primary measure might accidentally destroy. A validity check protects the evidence from misleading us.

When reality disagrees with our expectations, trace the evidence backward. Was the system built incorrectly? Was the selected shape inadequate? Or were the criteria or desired outcome mistaken? Each failure calls for a different correction.

The point is not to measure everything. It is to preserve the line between what we intended, what was built, and what actually changed. Evidence can challenge an accepted criterion. It should not silently redefine it.

Fit tells us what should work.
Evaluation tells us what does work.
Outcomes tell us whether it matters.

Sources and related methods

The synthesis and illustrative metrics here are proposed applications, not an official shared scorecard from these organizations.