Essay

Refuse Over Guess

September 2, 2026· 12 min read

The question nobody asks in the demo

Every AI system demo I have ever sat through answers every question it is asked. That is the point of a demo. Someone types a question, the system produces a fluent, confident, well-formatted answer, and the room nods. What the demo never shows is the question the system should not have answered: the one where the data was missing, the intent was ambiguous, or the model simply did not know, and it produced a fluent, confident, well-formatted answer anyway.

In a chat toy, that is a shrug. In a system that posts accounting entries, closes support tickets, or tells a plant manager how many units came off a line last quarter, a confident wrong answer is worse than no answer at all. A wrong number stated with conviction gets acted on. A refusal gets escalated to a person who can find the right one.

So the design principle I hold above every other in production AI work is simple to state and hard to build: the system must prefer to refuse rather than guess. It needs an explicit, engineered state for “I am not confident enough to answer this,” and that state has to route somewhere useful, usually a human, rather than being papered over by the model’s natural instinct to keep talking.

This essay is about what that principle looks like when you actually build it. I will draw on three systems I have shipped: a natural language query layer over a manufacturing data warehouse, a multi-agent customer support system, and an accounting reconciliation platform. They are very different products. The refusal machinery underneath them is nearly identical.

Why models guess

It helps to be precise about the failure. Large language models are trained to produce the most plausible continuation of their input. “Plausible” is not “true,” and it is definitely not “verified against your data.” When a model is asked something it cannot actually resolve, the plausible continuation is still an answer, because in the training data, questions are followed by answers far more often than by “I do not know.”

This is not a bug you can prompt away. You can ask a model to say “I am not sure” when it is not sure, and it will do so more often, but its sense of its own uncertainty is itself a plausible-sounding guess.¹ The model does not have privileged access to whether the number it just produced matches the database. It only has access to whether the number looks like the kind of number that appears in that position.

Once you accept that, the design question changes. You stop asking “how do I make the model know when it is wrong?” and start asking “what machinery, outside the model, decides whether this output is allowed to reach a user as a fact?” That machinery is where the real engineering lives.

A worked example: eighteen steps before an answer

The clearest version of this I have built is a system that lets a plant manager type a question in plain language about production data and get back a number they can defend in a production review meeting. The client was a manufacturing AI platform, the users were operations leaders, and the constraint was stated up front: the system would rather say it does not have a reliable answer than guess.

The pipeline runs through eighteen distinct verification steps before any answer is delivered. I will not walk through all eighteen, but the shape matters more than the count.

Authenticate and scope before you interpret

The first steps have nothing to do with language. The request is authenticated, and the user’s data access is resolved per user and per tenant. Only after the system knows exactly what this person is allowed to see does it start thinking about what they asked. That ordering is deliberate: if interpretation happens first, you end up with a query the model “understood” perfectly that touches data the user should never have reached, and now you are filtering results after the fact instead of never producing them.

Propose with a model, validate with rules

The question goes to a small language model whose job is narrow: propose an intent. Which metric, which time range, which grouping, which filters. That proposal is structured, not free text.

Then a fully deterministic rules layer independently validates every field of that proposal. Is this a metric that exists? Is this a time range that parses and falls inside the available data? Is this grouping valid for this metric? The rules layer does not consult the model and does not care how confident the model sounded. If any field fails validation, the pipeline stops there and tells the user which part of the question it could not resolve.

This split is the heart of the whole approach. The model does what models are good at, mapping messy human language onto a structured space. The rules do what rules are good at, refusing anything outside that space with zero tolerance. Neither is trusted to do the other’s job.

Prefer a provably correct path

Wherever possible, the underlying database query is generated through a compiled path: a fixed, tested translation from a validated intent into SQL, rather than asking a model to write SQL from scratch. A model writing SQL is fluent and frequently right. A compiler from a closed set of validated intents is right by construction. When the intent falls inside the space the compiler covers, the compiler wins. When it does not, that is a signal the system is at the edge of what it should be confident about.

Re-derive the answer from the raw data

After the query runs, the system does not hand the model the result and ask it to write a nice sentence. It re-derives the final answer directly from the raw returned rows and checks that the number it is about to state is mathematically consistent with them. A sum should equal the sum. An average should equal the average. This catches a whole class of errors where the query was fine, the data was fine, and the last-mile narration silently rounded, mislabeled, or hallucinated a figure.

Label every answer with a trust level

Finally, every answer carries an explicit trust level, so the user always knows how much confidence to place in what they are reading. “This came through the compiled path against validated fields” is a different statement from “this came through a fallback path with an inferred time range,” and the interface says so. A refusal is just the lowest rung of that same ladder: “I could not produce an answer I can stand behind, here is why.”

Security that can only narrow

One more detail from this system is worth calling out, because it is the same idea applied to permissions. Security is layered from tenant-level isolation down to row-level restrictions, and the layers are deliberately ordered so that each one can only narrow what a user is allowed to see, never widen it. That ordering is treated as a core guarantee, not an implementation detail. It is refuse-over-guess applied to access: when in doubt, show less.

The system shipped with more than seven hundred automated tests plus live validation runs against the client’s real production warehouse. I mention the test count not to impress but because it is the honest cost of the principle. Refusal machinery is a lot of code, and every step of it has to be tested, because a verification step that silently passes bad input is worse than no step at all.

The same principle in a support system

The manufacturing system is the heavyweight version. The same idea shows up in lighter form in a multi-agent customer support system I built for an internal operations team.

When a ticket arrives, the system searches a structured memory of past incidents, their root causes, and their resolutions. Then it branches on confidence. For well-understood, recurring issues that match memory closely, it generates and executes the resolution itself: updating the ticketing system and notifying the relevant team directly, with no human in the loop. That path is the “guess” path in the sense that the system acts on its own judgment, and it is only allowed because the judgment is anchored to a case it has seen resolved before.

For anything new, ambiguous, or unfamiliar, the system refuses to act. But it does not refuse uselessly. It hands the ticket to a human along with a clear, ranked summary of the most likely causes and the relevant supporting documentation already assembled. The refusal is a handoff package, not a dead end.

The result was that 70.2 percent of tickets are resolved fully automatically, and that number keeps rising as the memory grows. But the more interesting result was behavioral. The team began routing even simple tickets through the system, not for the automation, but because the assembled context was a better starting point than a blank ticket. A refusal that arrives with evidence is something people learn to want.

That is the second half of the principle, and it is easy to miss. Refuse over guess does not mean “do less.” It means that when the system cannot act, it should make the human’s job of acting as easy as possible. A system that only knows how to say yes will eventually act confidently on the wrong thing. A system that only knows how to say no will be switched off. The useful system knows how to say “not me, but here is everything you need.”

Where the line goes in an accounting system

The third system is an accounting and reconciliation platform that now processes more than a thousand transactions a day across more than ten client businesses. Here the refusal question is quantitative: what fraction of transactions is the system allowed to handle end to end, and what happens to the rest?

The answer landed between 70 and 90 percent fully automated, depending on the business, with the remainder flagged for a quick human check rather than a full manual review. Two design choices made that split work.

First, matching a bank transaction to an invoice does not rely on one signal. It fuses vector similarity, semantic matching, and counterparty identity using Reciprocal Rank Fusion, so no single method dominates the decision.² When the fused signals agree, confidence is high and the system proceeds. When they disagree, that disagreement is itself the refusal trigger.

Second, categorization across more than 140 expense categories runs as a three-stage pipeline: rule-based enrichment, an LLM classifier, then an LLM sanity-check pass. The final pass exists specifically to catch the classifier’s confident mistakes before they become journal entries. When a user corrects a categorization, the correction is captured and shapes future decisions without retraining anything.

The point is that “flag for human review” is not a failure state in this system. It is a designed output, sized deliberately, and instrumented like everything else: cost, latency, and accuracy are tracked per decision, so the client can see exactly how often the system refused and whether those refusals were warranted.

What refusal costs, and why it is worth it

I want to be honest about the price. Refuse-over-guess systems are slower to build than systems that just answer. They have more moving parts. They generate a stream of escalations that someone has to handle. And they will occasionally refuse a question a human could have answered in two seconds, which is annoying.

They are worth it for one reason: trust compounds and distrust compounds faster. A system that is right 95 percent of the time and confidently wrong the other 5 percent gets checked 100 percent of the time, because nobody knows which 5 percent they are looking at. That system has saved no one any work. A system that is right 90 percent of the time and says “not sure, here is why” the other 10 percent gets checked 10 percent of the time. That system has removed 90 percent of the work, and the 10 percent it hands back arrives with context.

The math only works if the refusals are trustworthy, which means the refusal machinery has to be at least as well engineered as the answer machinery. In most systems I see, it is an afterthought: a confidence threshold on a single model score, added late, tuned once. In the systems I build, it is the architecture.

A checklist

If you are designing a system that will act on model output, here is the shape I keep coming back to:

1. Scope before you interpret. Resolve who is asking and what they may see before any model reads the question.
2. Propose with a model, validate with rules. Let the model map language onto a structured space. Let deterministic code refuse anything outside it.
3. Prefer the provably correct path. Wherever a closed set of intents can be compiled rather than generated, compile.
4. Re-derive the answer from the data. Never let the last-mile narration be the only thing standing between the query result and the user.
5. Label the trust level. Make the confidence visible. Refusal is the bottom rung of the same ladder, not a separate error message.
6. Make refusals useful. A handoff should arrive with ranked hypotheses and assembled evidence, not an apology.
7. Instrument the refusals. Track how often the system declined, and whether it was right to.

None of this is exotic. It is the discipline of deciding, in advance and in code, exactly when the AI is allowed to speak with confidence and when it should defer to a person. The model is the least important part of that decision.

essaysagentic-aireliabilityverification