Skip to content

Reproducibility

The core claim: same scenario + same seed = same journal, same world, same digests — on any machine. CI proves it on every commit by comparing golden digests across 3 OS × 2 Python versions, byte for byte.

How determinism is constructed

  • Virtual clock. Time never comes from your machine: tools declare a virtual time_cost, scenarios schedule events at at_vt, and nothing advances implicitly. Wall-clock reads inside a run are blocked by a guard.
  • Keyed PRNG. Randomness is SHA256(seed ‖ domain ‖ counter) per domain — adding a new random draw in one domain never shifts another domain's sequence. Snapshots reduce PRNG state to a few counters.
  • Single writer + ordering contract. --determinism strict (default) admits concurrent tool calls in a deterministic key order. We guarantee execution determinism (what the admitted sequence does); we do not promise scheduling determinism of your agent's own concurrency.
  • Guards. During a run, sockets and real time are blocked (EM_GUARD) — zero telemetry is a testable property, not a policy.

Identity: repro_id

Content-addressed over the scenario content, seed, engine version, pack fingerprints (built-in and BYO), and base-snapshot digest. Two runs with the same repro_id are answering the same exam.

Seeds: fixed vs vary

eval-mock run s.yaml --repeat 10 --seed fixed -- <agent>   # pure agent variance
eval-mock run s.yaml --repeat 10 --seed vary  -- <agent>   # robustness across worlds

With --seed fixed the environment contributes zero variance, so pass-rate spread is your agent's alone — and aggregate.first_divergence points at the exact journal seq where a run first went its own way.

Third-party verification

Send someone the run directory; they run:

eval-mock verify runs/run-<id>

verify replays the journal in a fresh environment and re-derives the digests. Outcomes:

Outcome Meaning
VERIFIED the verdict reproduces exactly
SCENARIO_CHANGED scenario content differs from the manifest
ENGINE_MISMATCH different eval-mock version
PACK_CHANGED a pack fingerprint moved (incl. edited pack.yaml)
TRAJECTORY_DIVERGED replay differs — reported with the first divergent seq

Boundary, stated honestly: verify is environment-level replay (the same trajectory must yield the same verdict). It does not re-sample your agent — that is an evaluation-level question for --repeat.