Jev + SystemOne Tune in SuperQode (GEPA): teach decisions from unlabeled examples

We shipped the SystemOne harness for Jev-like models a few days ago. As soon as we got access to Jev we put progressive tool discovery into the harness: deferred optional tool schemas, a closed candidate set for tool_search, and shadow or rerank modes that let Jev recommend which schema to activate while SuperQode still owns permissions. One question was still open after that work landed. A reviewed pack can gate tools and route tickets today, but production decisions drift. How do we improve Jev's typed answers against our own unlabeled history without silently rewriting the pack the coding loop already trusts?

We have spent a lot of time with GEPA on prompt optimization and context optimization across Superagentic AI projects, so embedding GEPA into SystemOne was the natural next move. Keep the coding model on implementation. Keep SuperQode on control flow, redaction, and session evidence. Let GEPA search over the frozen decision pack that Jev evaluates: question instructions, Choice criteria, and the confidence policy that turns raw answers into typed outputs. We were mid-implementation when jev-align appeared on PyPI and GitHub, another GEPA-backed path for calibrating Jev functions from labeled judgment. Different product shape, same pressure: teams want production-shaped feedback into System One decisions. That was enough signal to ship this release as SystemOne Tune, documented in the SystemOne docs and surfaced on superqode.dev.

How GEPA optimizes Jev in Tune

A SystemOne pack is the unit of decision quality. It declares the input contract (state_schema, optional input_key), atomic questions (Choice, Score, or Noul), and a decision_policy with min_confidence, noul_false_max, and noul_true_min. The harness validates and redacts state, calls the selected client (live TypeSafe by default, or stub/replay), binds answers to their question types, and returns status, outputs, answers, abstained, and pack-hash metadata. Uncertain Choice/Score outputs become null and land in abstained; Noul abstains between the false and true thresholds. Jev stays the hosted decision model. GEPA never fine-tunes those weights. It proposes edits to the pack text and criteria that Jev will evaluate next, then scores those candidates against labels you reviewed.

Tune currently improves one fixed Choice question, including structured instructions and criteria, while preserving output labels, schema, model configuration, and confidence policy on the experiment. The active-learning path is the common case for production data: every imported label starts empty, Jev scores only the development pool each round, selects the most uncertain examples plus a random audit example, and draws a reserved test example without scoring the sealed test pool. You label a batch of two to twenty development judgments (default five) with optional rationales. Those labels feed GEPA's reflection model (OpenAI, Anthropic, Gemini, or a LiteLLM provider/model), which is a separate credential from TYPESAFE_API_KEY. GEPA proposes pack edits under a metric-call budget (--max-evals) and a reflection spend limit (--max-reflection-cost) checked between model calls. The harness writes candidate-pack.yaml, candidate-harness.yaml, changes.diff, evaluation evidence, and report.json under .superqode/tuning, with per-round artifacts in rounds/round-NNNN and accept/reject history in run.json.

Comparison counts abstentions and errors separately and checks for regressions against the baseline on the sealed set. Small runs (under 30 optimizer-visible development examples or 30 test examples) stay pilots and cannot qualify for verified adoption; you can still accept them as experimental after reading the report. Verified acceptance still needs representative volume, an improved sealed score, no individual regression, and no evaluation errors. Acceptance never rewires an active coding-session route. Rejected candidates leave the current pack unchanged and reuse cached pool predictions on the next round. That is the same GEPA loop we use for prompts and context, pointed at SystemOne packs, with SuperQode owning the accept gate.

Why Tune starts from unlabeled pools

Real routing and review traffic is rarely a clean labeled set with train/validation/test already carved out. You usually have a pile of past states (tickets, patch descriptions, tool-call contexts) and a sense that some of them sit near the decision boundary. Tune is built for that shape: point it at CSV, JSONL, JSON, or YAML with state / label columns (or map other names with --input and --label), or start from the checked-in examples/tune/factory-route-active.csv (35 development and 35 reserved routing inputs with blank labels). Optional id, rationale, group, and split fields keep related examples together and honor explicit train / validation / test or legacy held-in/held-out files. Duplicate inputs are rejected. Test examples never enter GEPA.

Fully labeled or partially labeled files keep the bounded one-shot workflow. Empty labels start the multi-round active-learning session described above. Either way, the result is staged as inspectable experimental output until you accept, reject, or leave it pending for --resume. Stop saves completed evidence; annotation and pending candidate decisions can resume, while an interrupted optimization requires a new experiment and budget so spend does not silently restart.

What you see in the product

From any SuperQode session open Tune in the TUI, or pick Improve decisions on the SystemOne entry in Harness Hub:

Or from the CLI after installing the optional, tested optimization runtime (TUI Install tuning support, or superqode harness tune --setup, then restart). That install pins a GEPA source revision into SuperQode's Python environment. Live Jev calls need TYPESAFE_API_KEY in the launching process. Reflection needs its own provider key. Missing keys fail with a clear message instead of looking like a no-op Start button. Coding-agent subscription credentials are not reused for reflection.

The free factory-route active-learning CSV is on GitHub:

Install or update SuperQode first if you need it:

Useful CLI knobs once you leave the demo path: --pack for a built-in pack, --spec for a decision harness, --output for the run directory, --reflection-lm to override the reflection model, --seed for deterministic splits, --batch-size for judgments per round, and --resume to continue annotation or resolve --accept --experimental / --reject from a script. Inputs go to Jev; development examples and rationales also go to the reflection provider. Existing state redaction applies, but example files should still avoid secrets.

Watch the walkthrough

The recording walks the end-to-end path: open Tune, load the unlabeled pool, review a Jev-ranked batch (uncertain, random audit, and sealed selections called out on the cards), run the GEPA experiment, and stop on an experimental candidate with the sealed comparison and exact question-pack diff on screen.

How this fits the SystemOne harness

Tune sits beside the other SystemOne surfaces rather than replacing them. Use superqode --harness systemone or :harness use systemone when you want Jev in the coding loop with shadow tool gates and deferred tool discovery (default tool_search_mode: shadow; set SUPERQODE_SYSTEMONE_TOOL_SEARCH=rerank when you want Jev to select which deferred schema to activate under the documented confidence and margin thresholds). Use :systemone connect <pack> or harness run for standalone typed decisions without a coding provider. Use :systemone live / shadow modes when you want native tool-permission checks or disagreement traces. Use :systemone tune when you want a better Choice pack from examples, with the active harness left alone until you promote a candidate.

The homepage on superqode.dev now leads the SystemOne section with Tune and embeds this demo. The full guide, including pack schema, labelled decision evals, rubric grading, and shadow disagreement reports, remains at docs.superqode.dev/advanced/systemone.

Try it

1. Install or update SuperQode (superqode update, or the curl installer above), then run superqode harness tune --setup and restart if tuning support is not already installed.
2. Export TYPESAFE_API_KEY and a reflection provider key (OpenAI, Anthropic, Gemini, or LiteLLM).
3. Download the factory-route CSV (or bring your own state/label columns with optional group and split fields).
4. Run :systemone tune or superqode harness tune --data ... --batch-size 5, read the experimental report and changes.diff, and accept only when the sealed comparison matches what you want the pack to do next.

Source is on GitHub. Packages are on PyPI. If you ship a pack this way, tell us what the uncertain examples looked like before labeling. That is the feedback Tune is designed around. Production-call capture, new-question discovery, Noul/Score tuning, and automatic adoption into permission or revision loops are still ahead; this release is the Choice-pack loop with an explicit human gate.