An agent without an evaluation harness is a demo

Everyone is building agents right now, and I understand why. Watching a model plan a task, call tools, check its own work, and come back with a finished result feels like the future arriving. On the right problem, it is.

Here is the uncomfortable part: every property that makes an agent impressive also makes it harder to trust. A single-turn model answers a question; you can score the answer. An agent takes actions in sequence, and each step depends on the ones before it. The space of things that can go wrong stops being a list and becomes a tree. A system that is right 95% of the time per step, across a ten-step task, finishes clean barely half the time. That arithmetic does not show up in a demo. It shows up in week three of real use.

So my rule, for my own work and for anyone who asks: an agent without an evaluation harness is a demo, no matter how good it looks. Before autonomy earns a place in a workflow that matters, I want to see task-level success rates measured on realistic cases, not cherry-picked ones; failure modes catalogued, especially the quiet ones where the agent finishes confidently with the wrong result; clear boundaries on what the agent is allowed to touch, enforced outside the model rather than requested politely in the prompt; and a human checkpoint anywhere an action is expensive to reverse.

None of that is exotic. It is the same test-and-evaluation discipline that every serious engineering field applies to systems that act on the world. The difference is that the AI field is currently sprinting, and evaluation is the part most teams skip when they sprint.

The teams that will still be running agents two years from now are the ones building the harness alongside the agent, not after it. Autonomy is not a feature you bolt on. It is a claim about reliability, and claims need evidence.