What Your AI Is Not Allowed to Decide
Safe AI in a real deployment is four operating rules about where the human sits. Here they are, with what each one costs and where the line gets drawn.
The question underneath "is it safe?"
Every operator asks some version of it before signing, and it almost never means what the vendor answers. The vendor hears "will the model hallucinate" and starts talking about accuracy rates. The operator means something narrower and much more practical: what is this thing allowed to decide without me?
That is a design question with a written answer, and it takes about an hour to settle per process. Here is ours, as four rules rather than a set of principles, because a principle cannot be checked on a Tuesday afternoon and a rule can.
Rule one: nothing binding goes out without a person
A price. A delivery date. A term. A document that a court would read as a commitment. All of it stops at a person.
This is the rule that survives contact with lawyers, and it is also the one that costs the least, because the binding step is a small fraction of any process. In a brokerage deployment, the AI answered first, qualified the enquiry, assembled the document package and routed it to the right agent. First response fell from 6 hours to 8 minutes and document preparation from 2 days to 20 minutes. What left the building still had a human name attached to the decision.
The six hours were queue time. Nobody was thinking for six hours; the enquiry was sitting in an inbox. Automation is very good at removing waiting, retyping and looking-up, and quite bad at carrying liability. Splitting those two is most of the design.
Rule two: exceptions arrive with the context attached
Any system that escalates uncertain cases to a human is spending that human's time by design. The variable is what shape the escalation arrives in.
An escalation that says "needs review" makes the approver rebuild the whole situation before deciding, and that reconstruction is usually longer than the decision. An escalation that arrives with the full conversation, the data the system used, and what it was uncertain about turns a five-minute reconstruction into a thirty-second judgement.
This matters financially, not just ergonomically. Escalation rate times approver time is a monthly cost that runs forever, and it belongs in the arithmetic before anyone signs rather than being discovered in month three.
Rule three: the system assembles, it does not compose
For anything contractual, the AI works from templates you have approved and data that has been validated. It fills, it selects, it arranges. It does not write new clauses.
This is a narrower promise than "the AI drafts your contracts" and it is the reason the legal review is short. A reviewer checking whether the right approved template was used with the right validated data is doing a fast, bounded job. A reviewer checking whether a generated paragraph creates an unintended obligation is doing a slow, unbounded one.
The same logic covers regulated content generally: administrative work runs unattended, professional judgement does not. In a clinic deployment, patient intake went from 45 minutes to 8, no-shows fell 40%, and administrative paperwork dropped by three quarters. Every clinical judgement stayed with a clinician, and nothing in the system was allowed to look like one.
Rule four: every automatic decision is on the record
If the system decided something on its own, there is a record of what it decided, on what input, and when. Not because anyone reads it routinely, but because the questions that eventually get asked are retrospective: why did this quote go out at that number, when did we start doing it this way, show me the ten cases before the complaint.
An audit trail also happens to be the evidence a regulated market asks for, which is why the compliance conversation is easier when the operational design came first. Documentation written to describe a system that already behaves well is honest. Documentation written to describe a system nobody constrained is aspiration with a cover page.
Where the line gets drawn
There is no universal answer, and any vendor offering one is selling a poster.
| Decision type | Runs unattended | Needs a person |
|---|---|---|
| Reminders, scheduling, routing, lookups | Yes | No |
| Qualification and triage | Yes, with a record | No |
| Anything priced, promised, or contractual | No | Always |
| Professional judgement in a regulated field | No | Always, by name |
| The grey middle | Decided per process | Decided per jurisdiction |
The grey middle is where the actual work is, and it gets settled during the diagnostic with the people who carry the liability, before anything is built. A clinic and a brokerage draw the line differently in the same country. The same clinic draws it differently in two countries.
Written down before go-live, that line is a design. Inferred afterwards from whatever the system happened to do, it is an incident report.
What we do not have
We do not publish a numbered list of principles, and this post is not one. What exists is the four rules above, applied across 200+ deployed workflows at 50+ companies, with the line redrawn per process during diagnostics.
If that sounds less impressive than a framework with six pillars, it is meant to. The useful thing about a rule is that somebody can check on a Tuesday whether it held, and then tell you what happened when it did not.
01Does a human approval step cancel out the speed you promised?+
It does not, because the slow part of most processes was never the decision. In a brokerage deployment first response went from 6 hours to 8 minutes and document preparation from 2 days to 20 minutes, and a person still signed off on everything that left the building. The six hours were queue time, not thinking time: the enquiry sat in an inbox until somebody got to it. Automation removes the waiting, the retyping, the looking-up and the assembling, and hands a person a finished thing to approve in a minute. That is a completely different use of the approver's attention than asking them to do the work. The cases where approval genuinely slows things down are the ones where the approver is a bottleneck already, and that shows up in the diagnostic as a queue in front of one person. If we find it, we say so, because automating up to a blocked approver just relocates the queue.
02Where exactly should the approval line sit?+
It is set per process and per jurisdiction, and there is no universal answer worth printing. The rule we apply is that anything binding on the company needs a person: a price quoted, a term agreed, a commitment made, a document that could be read as a contract. Anything purely administrative can run unattended: a reminder, a routing decision, a scheduling slot, a data lookup. The interesting cases sit between, and they get decided during the diagnostic with the people who carry the liability. A clinic and a brokerage in the same country will draw the line differently, and the same clinic in two countries will draw it differently again, because the regulation and the professional duty differ. What matters is that the line is written down before go-live and reviewed when the process changes, rather than being inferred later from whatever the system happened to do.
03What does the human-in-the-loop step actually cost to run?+
It costs the escalation rate multiplied by the approver's time, and it belongs in the business case as a monthly line item rather than as an assumption. If a process runs 400 times a month, escalates 12% of cases, and each escalation takes an approver 4 minutes, that is roughly 3 hours a month of a specific person's time, every month, forever. That is usually a good trade and it is never zero. Two things make it worse than it needs to be: an escalation that arrives without context, so the approver reconstructs the situation before deciding, and an escalation threshold nobody revisits after launch. We attach the full conversation and the reasoning to every escalation for the first reason, and we review thresholds during handover for the second. When you evaluate any vendor, ask for the escalation rate and the average approver time in the same breath as the payback number.
04How is this different from AI ethics and compliance work?+
Compliance work answers to a regulator and produces documents: risk classifications, model documentation, bias testing evidence, incident procedures. Human-in-the-loop design answers to the operator and produces behaviour: which decision goes where, who is accountable, what gets logged. The two overlap, because auditors ask for evidence that a human review step exists and that it is real rather than nominal, and an audit trail of automated decisions is exactly the evidence they want. But they fail differently. A company can be fully documented and still ship an automation that quietly commits it to a price it did not intend, and a company can have a sound approval design with none of the paperwork a regulated market will demand. Both need doing, and doing the operational half first tends to make the paperwork half honest rather than aspirational.
