The question with agentic systems is not whether they will make mistakes. They will. The question is what happens next, and that is a design decision you make before writing any code.

An agent that reads a document and drafts a summary carries almost no risk. An agent that reads a document, decides a claim is valid and posts a payment carries a great deal. Same architecture, entirely different design obligations — and the difference is not intelligence, it is consequence.

Classify actions by reversibility

Before designing anything, sort every action the agent can take into three buckets:

  • Freely reversible — drafting, tagging, suggesting, retrieving. Let the agent act autonomously.
  • Reversible with effort — creating records, sending internal notifications, updating non-financial fields. Autonomous above a confidence threshold, logged for review.
  • Effectively irreversible — payments, external communications, deletions, regulatory submissions. Always human-confirmed, regardless of confidence.

This classification does more for deployability than any amount of prompt engineering. It also gives risk and compliance a structure they recognise, which materially shortens approval.

Confidence has to be calibrated, not asserted

Language models are enthusiastic about their own conclusions. A raw self-reported confidence score is close to meaningless. Useful confidence comes from elsewhere: agreement between independent passes, retrieval quality, whether required fields were actually found in the source, and consistency with business rules.

We treat confidence as something the system earns from evidence, not something the model claims about itself.

Escalation must be cheap and specific

When an agent escalates, the human receiving it should get the case, the agent's partial work, the specific reason for escalation, and a one-click path to resolve. An escalation that dumps the whole problem back on a person destroys the efficiency the system was supposed to create.

Measure escalation quality, not just escalation rate. A system escalating fifteen percent of cases with useful context beats one escalating five percent that humans have to redo from scratch.

What we instrument from day one

Every tool call, every decision point, the evidence behind each one, and the full trace for any case a human touches. When something goes wrong — and it will — you need to be able to reconstruct exactly why.

Design the failure path first

What happens when the model provider is down? When the retrieval index is stale? When the agent loops? Each needs a defined behaviour, and "the queue backs up silently" is not one.

Our default is to fail closed and visible: the agent stops, the work routes to the existing manual process, and someone is told. Degrading gracefully into the old process is almost always better than degrading into confident wrongness.

Bound the blast radius

Rate limits, per-run spend caps, a maximum number of tool calls, and a hard stop on loops. These feel excessive during development and prove their worth exactly once — which is enough.