SuperGauge

💡What SuperGauge is

SuperGauge specifies a single artifact: the Agent Quality Record. One document, produced at the point where somebody decides whether an agent may reach production. It states what was measured, what it was measured against, which conditions had to hold, whether they held, who accepted the result, and which version to revert to.

Evaluation platforms grade individual runs. Control specifications constrain what an agent may do while it runs. Published evaluation frameworks close the development loop and are explicit that automated release decisions remain outside their scope. The decision itself currently leaves no durable trace, so a review months later depends on whoever still remembers it.

The format composes with those layers. A system already exporting OpenTelemetry traces can produce a valid record without changing runtimes.

🔍Why it exists

📊Sample record output

Digests throughout, so a third party can confirm that nothing changed between measurement and decision. The authority block is required: a record that identifies what an agent is, without recording what it was permitted to do, can be structurally valid and substantively wrong.

supergauge: "0.1"
profile: {id: sg/coding-agent, version: "0.1", tier: T2}

subject:
  agent: acme-support-triage
  harness_digest: sha256:9c1e...        # the whole loop, model included
  authority:                            # what it was permitted to do
    acs_policy_version: "2.1"
    sandbox: docker
    egress: deny-by-default

task_set:
  manifest_digest: sha256:7d02...
  held_out: 12
  sealed: true                          # closed to anything that tunes

measures:
  - {id: task.completion,        value: 0.83, split: held-out, n: 12}
  - {id: reliability.pass_hat_k, value: 0.66, k: 5}

gates:                                  # deterministic; a judge cannot satisfy one
  - {id: policy.no_pii_egress,   result: pass, source: acs}
  - {id: reliability.pass_hat_k, floor: 0.60, result: pass}

decision:
  verdict: ship
  actor: "priya@acme.example"
  rolls_back_to: sha256:2f7a...
  signature: ed25519:...

🔄How a record is produced

Five stages. Each one contributes a field that a later reader can verify independently.

1. 1

   Define

   A profile sets the tier, the mandatory checks and the thresholds they must clear.

2. 2

   Seal

   The held-out portion of the task set is fingerprinted and closed to anything that tunes.

3. 3

   Measure

   The agent runs k times under a recorded authority grant, producing an event ledger.

4. 4

   Evaluate

   Deterministic checks and model-graded scores are computed from that ledger.

5. 5

   Record

   A signed record states what held, who accepted it, and the version to revert to.

📐The measure registry

The specification is small and changes rarely. The set of things worth measuring is neither, so the two are versioned separately. Each measure declares whether it is computed deterministically or graded by a model, because only a deterministic measure may block a release.

Measure identifiers are permanent. Where the meaning of a measure changes it receives a new identifier, so records already issued remain interpretable.

🚦The release rule

⚖️Tiers

Applying identical controls to a documentation fix and a payments change produces process that teams quietly route around. A profile declares a tier, and the tier determines which checks are mandatory. A profile may always demand more than its tier requires.

🪜Conformance levels

Level 4 addresses a specific failure mode. An immutable record guarantees only that whatever was submitted remains unchanged. Independent reproduction from the underlying ledger is what establishes that the submitted result was accurate in the first place.

🔌Interop and emitters

SuperGauge is not an eval runner. Peer tools keep grading and tracing; an emitter serialises values they already hold into an Agent Quality Record afterward.

🤝Contributing

The specification is governed narrowly and the registry is open. The range of behaviour worth measuring exceeds any single team's experience, and the practitioners who know what a sound test looks like are frequently not the people who write specifications.

🌍Adoption

An open invitation

📋Status and scope