SuperGauge

What is SuperGauge

SuperGauge specifies a single artifact: the Agent Quality Record. One document, produced at the point where somebody decides whether an agent may be released. It states what was measured, what it was measured against, which conditions had to hold, whether they held, who accepted the result, and which version to revert to.

Evaluation tools provide results and runtime controls record permissions. AQR connects those inputs to a versioned decision that reviewers can inspect.

The format composes with those layers. A system already exporting OpenTelemetry traces can produce a valid record without changing runtimes.

Why it exists

Use it with your existing stack

Start with one agent and the evaluation tools your team already uses.

1. Agree the release criteria and name the approver.
2. Evaluate against a versioned dataset with a protected held-out split.
3. Emit an AQR with the results, permissions and evidence references.
4. Validate conformance and check the recorded release approval.

super agent evaluate developer --gauge-out record.yaml

sq gauge gate record.yaml --level L2 --require-ship --quiet

Sample record output

Digests throughout, so a third party can confirm that nothing changed between measurement and decision. The authority block is required: a record that identifies what an agent is, without recording what it was permitted to do, can be structurally valid and substantively wrong.

supergauge: "0.1"
profile: {id: sg/coding-agent, version: "0.1", tier: T2}

subject:
  agent: acme-support-triage
  harness_digest: sha256:9c1e...        # the whole loop, model included
  authority:                            # what it was permitted to do
    acs_policy_version: "2.1"
    sandbox: docker
    egress: deny-by-default

task_set:
  manifest_digest: sha256:7d02...
  held_out: 12
  sealed: true                          # closed to anything that tunes

measures:
  - {id: task.completion,        value: 0.83, split: held-out, n: 12}
  - {id: reliability.pass_hat_k, value: 0.66, k: 5}

gates:                                  # deterministic; a judge cannot satisfy one
  - {id: policy.no_pii_egress,   result: pass, source: acs}
  - {id: reliability.pass_hat_k, floor: 0.60, result: pass}

decision:
  verdict: ship
  actor: "priya@acme.example"
  rolls_back_to: sha256:2f7a...
  signature: ed25519:...

How a record is produced

Five stages. Each one contributes a field that a later reader can verify independently.

1. 1

   Define

   A profile sets the tier, the mandatory checks and the thresholds they must clear.

2. 2

   Seal

   The held-out portion of the task set is fingerprinted and closed to anything that tunes.

3. 3

   Measure

   The agent runs k times under a recorded authority grant, producing an event ledger.

4. 4

   Evaluate

   Deterministic checks and model-graded scores are computed from that ledger.

5. 5

   Record

   A record states what held, who accepted it, and the version to revert to. Signing is required by higher assurance levels.

The measure registry

The specification is small and changes rarely. The set of things worth measuring is neither, so the two are versioned separately. Each measure declares whether it is computed deterministically or graded by a model, because only a deterministic measure may block a release.

Measure identifiers are permanent. Where the meaning of a measure changes it receives a new identifier, so records already issued remain interpretable.

The release rule

Tiers

Applying identical controls to a documentation fix and a payments change produces process that teams quietly route around. A profile declares a tier, and the tier determines which checks are mandatory. A profile may always demand more than its tier requires.

Conformance levels

Level 4 addresses a specific failure mode. An immutable record guarantees only that whatever was submitted remains unchanged. Independent redeployment from the underlying ledger is what establishes that the submitted result was accurate in the first place.

Interop and emitters

Evaluation tools run tests and store traces. An emitter writes those results into an Agent Quality Record.

Contributing

The specification is governed narrowly and the registry is open. The range of behaviour worth measuring exceeds any single team's experience, and practitioners can contribute test definitions from their own domains.

Adoption

An open invitation

Status and scope