A system that answers everything is not a confident system. It is an unmeasured one. The interesting engineering in an AI feature is usually not in making it answer — it is in deciding when it should not, and what happens next.
That decision belongs to the product before it belongs to the model.
Three different things called "refusal"
They need different handling and get conflated constantly.
Out of scope. The question is reasonable but the corpus has nothing to say about it. Correct response: say so plainly, and offer the route to someone who does know.
Out of bounds. The question asks for something the system must not do regardless of what it knows — commit to a price, give medical or legal advice, act on another customer's account.
Low confidence. The corpus probably contains the answer but retrieval is weak or the sources disagree. This is the one worth the most care, because it is the one where a confident wrong answer is most likely and most damaging.
Only the second is a policy question with a fixed answer. The first and third are judgement calls with a threshold, and a threshold is a product decision about which error you would rather make.
Setting the threshold honestly
Ask the question in business terms: what does a wrong answer cost here, and what does an unnecessary handoff cost?
For an internal knowledge assistant, a wrong answer costs a few minutes and some trust; an unnecessary handoff costs a colleague's attention. Threshold low, answer more.
For anything a customer will act on — a cancellation window, a price, an eligibility rule — a wrong answer costs a refund, a complaint, or a regulator. Threshold high, hand off sooner. In AI Trainer the refusal rate on the should-refuse bucket sits around 94% and we treat a drop there as a release blocker, while the same behaviour on in-scope questions would be a bug.
The number itself is uninteresting. What matters is that someone chose it deliberately, wrote down why, and can see it move.
Refusing well is a writing problem
Most refusals are bad not because they refuse but because of how they read. Three rules:
Say what you do not know, not what you cannot do. "I do not have the 2026 cancellation terms for this supplier" is useful. "I am unable to assist with that request" is not — it tells the user nothing about whether to rephrase, look elsewhere, or ask a person.
Always offer the next step. A refusal that ends the conversation converts a small failure into a dead end. Attach the handoff: the person, the queue, the document, the form.
Never apologise twice. One acknowledgement, then the useful part. Padding makes a system feel evasive rather than careful.
The handoff is the actual feature
Routing to a human is where most implementations get thin, and it is where the user's experience is decided.
Carry the context. The person picking this up should see the question, what was retrieved, the confidence, and why it stopped. Making them start from "how can I help?" wastes the work and irritates the customer twice.
Close the loop. When the human answers, that answer is a candidate for the corpus or the regression set. A system where handoffs feed back improves; one where they vanish into an inbox stays exactly as good as the day it launched.
And measure the handoff, not just the refusal. The number that matters is not how often the system declined — it is how often a declined question was answered well by someone, and how long that took. A guardrail that protects the model's accuracy while quietly degrading the customer's experience has not solved anything, it has moved the failure somewhere it is harder to see.