Agent Quality
Set the checks an agent must pass before release.
We help your team test an agent and document its release decision. The work starts with your current runtime, evaluation tools and incident history. Agent testing extends software testing with repeated runs, model judge calibration and tool-policy checks.
- Task set: drawn from your workload, held-out split sealed
- Run k times: independent repeats, working state reset between each
- Deterministic checks: the only measures allowed to block a release
- Model judge: reported in the record, never satisfies a gate
- Gates: pass or fail, no averaging
- Record: ship or hold, with the way back attached
Your workload
Cases come from the tasks your agent handles and the failures your team needs to catch.
Release approval
Passing checks and a recorded approval are required before deployment.
Client ownership
Your engineers receive the suite, pipeline configuration and operating procedure.
Starting scope
One agent, one evaluation path and one CI pipeline.
Typical duration
Two to three weeks once access and a domain reviewer are available.
Client input
A sandbox, representative tasks, an engineering owner and a release approver.
Fit
Who this helps
Quality teams
You need coverage for variable answers, tool calls and multi-step tasks.
AI platform teams
You need a repeatable policy for prompt, model and tool changes.
Engineering leaders
You need evidence explaining why a release was accepted.
Process
How we work
Define the criteria
Agree failure cases and thresholds with your domain reviewer before testing.
Establish a baseline
Version the dataset, protect the held-out split and calibrate judges against human labels.
Connect the pipeline
Run checks in CI. Exercise failed and held candidates to confirm deployment is blocked.
Transfer the workflow
Train the owner to rerun evaluations, review evidence and add regression cases.
Deliverables
What you receive
Evaluation dataset
Versioned cases, rubrics and a held-out manifest
CI policy
Deterministic checks and a separate approval requirement
Agent Quality Record
Baseline and candidate evidence linked to the decision
Runbook
A documented procedure and a trained client owner
Scope
Engagement Options
We quote after reviewing the agent, available evidence and access requirements. Pilot timing depends on those inputs. Model and infrastructure costs are agreed separately.
Typical starting point
Agent Release Quality Pilot
Begin with one agent
Typically 2 to 3 weeks
One agent and one CI pipeline
Install a repeatable evaluation and approval workflow in your repository.
- Dataset and baseline results
- Deterministic release checks
- AQR emission and validation
- Failed-candidate demonstration
- Owner training and handover
Evaluation System
Extend the coverage
Typically 4 to 6 weeks
Additional agents and failure cases
Expand datasets, repeated-run testing and adversarial coverage around an agreed baseline.
- Tool and permission tests
- Injection and recovery scenarios
- Judge calibration
- Regression maintenance procedure
Quality Support
Maintain the workflow
Release reviews and dataset maintenance
Agree a bounded schedule for reviewing changes and adding cases from incidents.
- Release evidence reviews
- Calibration updates
- Dataset changes
- Engineering support
Open source
Underlying systems
Keep your runtime and observability tools. We agree the integration during scoping.
Questions
Questions
Scope an Agent Release Quality Pilot
Tell us which agent you want to test, what evaluations you run today and who owns the release.
London and San Francisco. Remote or on site.