Skip to content

Output contracts

Two files are the downstream contract for CI parsers, judges, and tooling. Both carry schema_version; minor releases only add optional fields. The compatibility promise covers three things: scenario schema, error codes, and the canonical digest format.

Journal (journal.jsonl)

Append-only, one JSON object per tool call — exactly these 14 fields:

Field Meaning
seq 1-based position in the journal
arrival_seq request arrival order (concurrency-visible)
commit_seq commit order (== seq)
turn_id scripted-turn the call belongs to
vt virtual time (ISO-8601) at call start
tool canonical dotted tool name
args validated arguments
result success payload (null on error)
error error envelope (null on success)
fault_injected true if a declared fault fired
world_version_before store version before the call
world_version_after store version after the call
wall_start_ns wall-clock start — diagnostics only, excluded from digests
wall_end_ns wall-clock end — diagnostics only

World events applied by the clock appear as tool: "world.event" entries.

Verdict (verdict.json)

Run-set shaped from day one — n=1 is just a special case, so CI parsers never migrate:

schema_version, digest_version, repro_id
runs[]: run_id, seed, status, steps, final_response,
        journal_digest, world_digest, world_diff,
        assertions[]: id, kind, pass, ring, intermediate, journal_refs,
                      weight, severity, error
aggregate: n, pass_rate, per_assertion{id: rate}, first_divergence?
  • status ladder (fail-closed): budget_exceeded > invalid (never called eval.finish) > fail (a critical assertion is false) > pass.
  • ring on failed assertions is the earliest failing ring: no-op/abandoned → tool-enabled → args-valid → order-valid → final-state → collateral, with journal_refs pointing at the evidence.
  • world_diff is the per-run world change tree (added/removed/changed per collection) — the data behind eval-mock diff.
  • first_divergence (repeat runs): first journal seq where a run differs from run 1, wall noise excluded.

Digests

digest_version: "1". Canonicalization is RFC 8785 (JCS); journal digest is a SHA-256 chain over wall-stripped entries; world digest hashes the store's canonical iteration; run = SHA256(journal ‖ world). repro_id is content-addressed over: scenario content, seed, engine version, pack fingerprints, base-snapshot digest.

The run directory is self-contained: manifest.json, verdict.json, journal*.jsonl, scenario.yaml copy, BYO pack.yaml copies, subject.json (who was tested — recorded faithfully, never interpreted).