Key takeaways
- A demo proves the happy path exists; production value lives in the edge cases and escalation logic.
- Start from the decision you're automating and its cost of error — not from the model.
- Constrain agents aggressively: grounded data, few well-tested tools, structured outputs.
- Human-in-the-loop by design — an agent that knows when to escalate beats one that bluffs.
- Instrument everything and ship narrow: measurable value first, expansion second.
The demo trap
Every week we see a slick AI agent demo: it books a meeting, drafts an email, answers a question in seconds. The room applauds. Then the thing hits production and quietly falls apart — it hallucinates a price, forgets context after three turns, or stalls because a real customer phrased their request in an unexpected way. The gap between a demo and a dependable agent is where most AI budgets go to die.
We build agents for a living, and our starting assumption is uncomfortable but useful: a demo proves nothing except that the happy path exists. Real value lives in the unhappy paths, the edge cases, and the boring operational plumbing nobody films.
Start from the decision, not the model
The first question we ask is never "which model?" It's "what decision or task are we trying to remove from a human's plate, and what does it cost when that decision is wrong?" An agent that drafts internal notes can be loose and creative. An agent that quotes prices, approves refunds, or talks to customers cannot.
Anchoring on the decision gives us three things immediately:
- A tolerance budget — how much error is acceptable before the agent must hand off to a human.
- A value metric — hours saved, response time, conversion, error rate — defined before we write a line of code.
- A scope boundary — what the agent explicitly will not attempt.
If we can't name the metric, we don't build the agent. That single discipline kills more vanity projects than any technical review.
Constrain aggressively, then loosen
The instinct with LLM agents is to give them freedom and tools and hope emergent intelligence does the rest. In production, freedom is risk. We design agents like we design good employees: clear role, clear boundaries, clear escalation.
In practice that means grounding answers in your actual data through retrieval rather than model memory, giving the agent a small set of well-tested tools instead of a sprawling toolbox, and forcing structured outputs wherever a downstream system consumes the result. A refund agent should return a decision object your ERP can act on — not a paragraph of prose someone has to re-key.
Humans in the loop, by design
The most valuable agents we've shipped aren't fully autonomous. They handle the 80% of cases that are routine and confidently escalate the 20% that aren't. That confidence threshold is a product decision, and it's where trust is earned. An agent that knows when to say "I'm not sure, routing this to a person" is worth far more than one that bluffs.
Instrument everything from day one
You cannot improve what you can't see. Every agent we deploy logs its reasoning inputs, tool calls, and outcomes so we can measure real-world performance, not lab performance. This does two things: it lets us catch drift and failure patterns early, and it turns the agent into a compounding asset. Each week of logs is training data for the next iteration.
This is also where cost discipline lives. Token costs, latency, and tool-call volume all show up in the numbers. We've cut client running costs by half simply by routing simple queries to cheaper models and reserving the expensive reasoning for cases that genuinely need it.
Ship small, prove value, then expand
Our happiest projects started narrow: one workflow, one team, one measurable outcome. A support triage agent. An internal knowledge assistant. A lead-qualification flow. Once it demonstrably saves time and people trust it, expansion is easy — the organisation pulls the agent into new areas rather than us pushing it.
The failed projects almost always tried to boil the ocean: a single "do everything" assistant that did nothing reliably. Scope is the difference between an agent that becomes indispensable and one that becomes a punchline.
What good looks like
A production-grade AI agent, in our experience, is boring in the best way. It does a defined job, most of the time, without drama. It escalates gracefully. Its performance is visible on a dashboard. It costs a known amount per transaction. And its value is expressed in numbers a CFO recognises — not in the applause it got in a meeting six months ago.
That's the bar. Demos are theatre; agents are infrastructure. Building the second kind takes less magic and more engineering discipline than most people expect — which is exactly why the results last.
If you're weighing where AI agents could genuinely move the needle in your operations, we're always happy to talk it through — no pitch, just a straight conversation about what's worth building and what isn't.
