Scenario DSL

One YAML file = one self-contained, distributable evaluation (schema_version: "1.0"). Unknown fields fail closed with a nearest-field suggestion; every load error carries a code / message / hint / location envelope with the YAML line and column.

Top level

Field	Required	Purpose
meta	yes	identity: name, schema_version, revision, seed, start_time
task	yes	what the agent is asked to do + budgets
world	yes	initial entities per pack (+ base_snapshot, from_file)
tools	yes	enabled packs: built-in names and/or pack.yaml paths
faults	no	deterministic fault injection
turns	no	scripted multi-turn follow-ups
assertions	yes	what "pass" means

task

Exactly one of prompt (string) or messages (list of {role, content}), plus budgets: max_steps (tool calls) and max_duration (virtual time, e.g. "1h", "90m"). Exceeding a budget ends the run with BUDGET_EXCEEDED and exit code 3.

world

world:
  base_snapshot: base.sqlite      # optional: restore instead of seeding
  email:
    inbox:
      - { from: boss@acme.example, subject: "Q3", body: "..." }
  shop:
    from_file: fixtures.jsonl     # optional: bulk rows, one JSON object per line
    orders:
      - { status: delivered, total: 42.0 }

from_file lines are {"collection": "...", "data": {...}}; paths are relative to the scenario file. base_snapshot files come from eval-mock snapshot and load an order of magnitude faster than re-seeding large worlds. Both are part of the reproduction identity.

faults

faults:
  - tool: email.send
    error: timeout            # timeout | rate_limit | server_error
    after_calls: 1            #   | permission_denied | malformed
  - tool: crm.search_contacts
    error: rate_limit
    probability: 0.2          # drawn from the keyed PRNG "faults" domain

Exactly one trigger per fault: after_calls: N (the Nth call fails, once) or probability: p (reproducible per seed). Injected calls are journaled with fault_injected: true.

turns

turns:
  - user: "Actually, please move it to 15:00."
    when: journal.count("calendar.create_event") >= 1     # CEL over the journal
    world_events:
      - at_vt: "+8s"                                       # or absolute ISO time
        email.deliver: { from: cfo@acme.example, subject: "Re: ..." }

Pull-based: the agent calls eval.next_turn to receive the next scripted user message; a when: that is not yet satisfied returns TURN_NOT_READY (journaled). Zero LLM at runtime — turn content is pre-written, triggers are deterministic.

assertions

assertions:
  - id: refunded
    expr: world.shop.orders.exists(o, o.id == 2 && o.status == "refunded")
  - id: retried
    expr: journal.calls("email.send").filter(c, c.fault_injected).size() == 1
    severity: minor            # minor failures never block the gate
    weight: 0.5
  - id: no-collateral
    collateral:
      allow_changes: ["shop.orders", "shop.refunds"]

Each assertion has exactly one of expr (CEL) or collateral (world-diff allowlist). The CEL context is world.* (final state), diff.* (added/removed/changed per collection), and journal.* (entries, tools, final_response) plus helpers calls / count / before / last / all_of / any_of. Prefer diff.* and exists() over positional indexing — eval-mock lint flags fragile patterns (W001–W004, see Error codes).