Agent Quality
An agent mishandles a customer. Someone asks what was checked before it went live.
Agents fail differently from software. One succeeds four times in five, takes a different route on every run, and reports its own work as finished. Conventional QA has little purchase on that, and the discipline that does is mostly a setup problem: once the tests, the release conditions and the reporting exist, the team you already have can run them. This engagement puts that in place, built from the way your systems break, and hands it over.
Built on what broke
Tests come from incidents your systems have already had, and from the work they do every day.
Objective release conditions
Blocking checks resolve to pass or fail. Model-graded scores sit alongside them and inform the judgement.
Handed over
The suite, the gates and the written procedure transfer to your engineers when the engagement closes.
Task set
drawn from your workload, held-out split sealed
Run k times
independent repeats, working state reset between each
Deterministic checks
the only measures allowed to block a release
Model judge
reported in the record, never satisfies a gate
Gates
pass or fail, no averaging
Record
ship or hold, with the way back attached
Fit
Consider this if any of these sound familiar
Teams tend to arrive with two or three of them at once.
Misbehaving agents
It picks the wrong tool, answers from nothing, or takes a different route every run, and nobody can say why.
Evaluations that miss
The suite passes, the customer still gets hurt, and the scores carry no weight in the room where releases are decided.
An agent you cannot maintain
It shipped and it worked. A model version, a prompt edit or a new tool has since moved it, and holding it steady is a full-time argument.
No gate before production
Changes reach customers on someone’s judgement under deadline, with no written condition an agent has to clear first.
Scope
Ten ways an agent goes wrong, and what we do about each
Most teams recognise three or four of these from the past quarter. Every one of them is measurable, and the measurement is what turns an argument about anecdotes into a release decision.
| The problem | What we do about it |
|---|---|
| The agent reports success. The work never happened. | Completion is judged against the state of your systems afterwards: the ticket, the row, the branch, the booking. An agent that writes a confident summary over an unchanged database is recorded as a failure. |
| It reaches the answer by a different route every run. | The execution path is graded for required steps, repeated loops and the step budget. Wandering shows up as a number in the report before it shows up on the invoice. |
| It picks the wrong tool, or calls the right one five times. | Every tool call is checked for choice, argument validity and repetition, so over-calling and missed calls are counted as defects instead of dismissed as noise. |
| It answers confidently with nothing behind it. | Claims are traced back to the retrieved context or tool output supporting them, and unsupported statements are counted per run. |
| Fine for ten minutes, then it falls apart. | Quality is measured across the length of a session, so degradation that only appears at turn thirty becomes visible before a customer finds it. |
| It worked when we demoed it. | The same task is run repeatedly under identical conditions, which separates an agent that works from one that worked once. This is usually the most uncomfortable figure in the first report. |
| Nobody knows what it would do if someone tried. | Prompt injection, tool misuse, privilege escalation and data-scope breach run as a versioned suite on every change, so the answer stops being a matter of opinion. |
| The bill went up and nobody can say what it bought. | Spend, tokens and elapsed time are reported per completed task, which distinguishes an agent that succeeds expensively from one that succeeds well. |
| Nobody can say whether it is ready for production. | Written conditions an agent has to clear before release, enforced as a pipeline step that fails the build when one is unmet. Readiness becomes a check with a result attached, applied the same way to every change. |
| You built the agent. Keeping it working is a different job. | A suite that runs on every model bump, prompt edit and tool change, so a regression is caught by the pipeline instead of by a customer. Your existing engineers operate it; the routine for extending it is written down and handed over. |
Process
How we work
Agree the bar
We work through your agents, incident history and exposure, then write down what each one has to satisfy. Criteria are settled before measurement starts.
Build the tests
Cases drawn from daily work and past failures, split into a development portion and a held-back portion that is fingerprinted and closed to tuning.
Put the gates in the pipeline
Blocking checks wired into CI, adversarial coverage added, and consistency established across repeated runs to give you a defensible baseline.
Hand it over
Your engineers run and extend the suite. We document the routine, train a named owner, and leave the whole thing in your repository.
Deliverables
What you receive
Test suite
Versioned cases from your own workload, with the held-back portion sealed and fingerprinted
Release gates
Blocking checks and adversarial coverage running in your pipeline on every change
Release record
A document per release stating what was verified, what held, and who accepted it
Operating routine
The written procedure for keeping the suite current, and a trained owner on your side
Scope
Engagement Options
Three columns, scoped and bought independently. Most engagements begin with the first. Scope and cost are quoted against your systems once we have seen them, since the size of the task set follows the number of agents and the shape of your failures. Architecture and instrumentation sit under Agent Engineering, and improvement against an established baseline under Agent Optimization.
Typical starting point
Evaluation System
Working out what good means
Typically 4 to 6 weeks
Criteria, tests and CI
The measurement foundation: written criteria, a test suite built from your own work and failures, and grading that looks at outcomes.
- Criteria written for each agent role
- Test suite from your workload and incidents
- Held-back portion sealed and fingerprinted
- Outcome, execution-path and tool-use grading
- Blocking and advisory checks running in CI
Assurance & Reliability
Working out what must hold
Typically 4 to 6 weeks
Adversarial coverage and consistency
The conditions under which an agent causes damage, and the consistency measurement that a single good run hides.
- Direct and indirect prompt injection coverage
- Tool misuse and privilege escalation testing
- Permission and data-scope boundary tests
- Consistency baseline across repeated runs
- Fault injection and recovery behaviour
Release Evidence
Working out what you can show
Typically 4 to 6 weeks
Records and operating model
The documentation layer: a record for each release decision, promotion criteria, a defined way back, and the routine that keeps everything current.
- Release record produced by your pipeline
- Promotion criteria and documented rollback
- Judge calibration against a human sample
- Procedure for turning incidents into tests
- Named owner trained on your side
Open source
Underlying systems
Our tooling is open source and the evidence format is a published specification. Engagements run against the stack you already operate, and adopting either remains your choice.
Questions
Questions
Discuss Agent Quality
Tell us what your agents do, what went wrong most recently, and what you would want to see before signing off a release. We come back with scope.
London and San Francisco. Remote or on site.