Comparison — and when not to use us
Honest positioning. eval-mock is a deterministic simulated world for agent evaluation; most tools below solve adjacent problems well.
vs API mocks (WireMock, Postman mocks, respx…)
| API mocks | eval-mock | |
|---|---|---|
| State | canned responses per endpoint | one shared world; every tool sees every change |
| Cross-tool effects | none (email "sent" is invisible to CRM) | first-class (delivery, referential integrity) |
| Time | real clock | virtual clock, schedulable events |
| Reproducibility | request-level | run-level canonical digests + third-party verify |
Use an API mock when you're testing client code against one API. Use eval-mock when the subject is an agent acting in a world.
vs benchmarks (τ-bench-style suites)
Benchmarks ship fixed tasks and a leaderboard; eval-mock ships the environment engine so you write exams about your domain, inject your failure modes, and gate your CI. Benchmarks answer "how do models compare?"; eval-mock answers "did my agent regress this week?" (and packs enough determinism that the diff pinpoints the step where it did).
vs eval platforms (DeepEval, Langfuse, LangSmith…)
Complementary, not competing — that's why we ship adapters. Platforms bring metrics, judges, tracing UIs; they typically replay recorded traces or call your live tools. eval-mock supplies the missing piece: a stateful, resettable, deterministic world for the agent to act in, plus evidence (journal, diff, rings) their judges can consume.
When not to use eval-mock
- You need real-API conformance. Our packs are honest simulations, not bug-for-bug replicas of Gmail or Salesforce. Contract tests against staging still matter.
- Code-execution benchmarks. SWE-bench-style "run the repo's tests in a container" is a different, well-served problem.
- Free-form conversation quality. Runtime is zero-LLM by design; user turns are scripted. If you need a simulated user with judgment, pair us with an external judge — or use something else.
- Single-shot prompt evals. If no tools and no state are involved, a plain eval framework is simpler.