Skip to content

Quickstart

Everything on this page runs offline: no API key, no account, no network.

1. First verdict (seconds)

pip install eval-mock
eval-mock quickstart

This runs the built-in email-basic preset with a scripted demo agent and prints a passing verdict plus a run directory. The same command is timed in our CI on a cold machine — install to first verdict must stay ≤ 60s.

2. Same world, your agent

Over MCP (your agent unchanged — it receives the server URL in EVAL_MOCK_MCP_URL):

eval-mock run --preset email-basic -- <your agent command>

Or native function calling, ~10 lines in your own loop:

from eval_mock.integrations.env import EvalMockEnv

env = EvalMockEnv("scenario.yaml")  # or a preset file
tools = env.openai_tools()  # or env.anthropic_tools()
# ... send tools with your model call; for each tool_call the model makes:
message = env.call(tool_call)  # executes + returns the reply message
env.finish(final_answer)
env.assert_pass()  # or env.verdict() for the full JSON

Tool names are sanitized for providers (email.send → email__send) and mapped back automatically — see Integrations.

3. Make it a gate

eval-mock run --preset email-gate --repeat 5 --pass-threshold 0.8 -- <agent>
eval-mock run scenario.yaml --null-agent    # mutation test: must NOT pass

Exit codes: 0 pass · 1 fail · 2 invalid (fail-closed) · 3 budget exceeded.

4. Prove it happened

eval-mock verify runs/run-<id>     # third-party replay of the exact verdict
eval-mock diff runs/run-<id>       # what changed in the world
eval-mock compare runs/A runs/B    # did behaviour change between two runs?

Next