Integrations

Five ways in; all evaluate against the exact same world and produce the same verdict files. Runnable versions of everything below live in examples/.

MCP (agent unchanged)

eval-mock run scenario.yaml -- <your agent command>   # URL in EVAL_MOCK_MCP_URL
eval-mock serve scenario.yaml --transport http        # long-lived, watch & iterate

Streamable HTTP + stdio, official SDK, pinned mcp>=2.0,<3. Run lifecycle: call eval.finish with your final answer (it's assertable); budgets end the run with exit code 3. Concurrent tool calls execute in a deterministic order under --determinism strict (default).

OpenAI function calling

env = EvalMockEnv("scenario.yaml")
tools = env.openai_tools()
# response = client.chat.completions.create(model=..., messages=msgs, tools=tools)
for tool_call in response.choices[0].message.tool_calls:
    msgs.append(env.call(tool_call))  # {"role": "tool", ...} ready to append

Anthropic tool use

tools = env.anthropic_tools()
# response = client.messages.create(model=..., messages=msgs, tools=tools)
results = [env.call(block) for block in response.content if block.type == "tool_use"]
msgs.append({"role": "user", "content": results})

Provider tool names cannot contain dots, so email.send is exposed as email__send and mapped back on dispatch. Hallucinated tool names still reach the journal as TOOL_NOT_ENABLED — visible to assertions. Unparseable arguments come back as MALFORMED_TOOL_CALL without touching the world.

pytest (evals as tests)

def test_my_agent(eval_mock_env):  # fixture ships with the package
    env = eval_mock_env("scenarios/email.yaml")
    ...
    env.finish("done")
    env.assert_pass()  # raises with ring attribution

JUnit for CI dashboards: pytest --junitxml=report.xml. A copy-paste GitHub Actions gate lives at examples/ci/github-actions-eval.yml.

DeepEval (optional extra)

pip install 'eval-mock[deepeval]'

from eval_mock.integrations.deepeval import to_deepeval_test_cases

cases = to_deepeval_test_cases(verdict, task_prompt=prompt)

Division of labour: eval-mock records deterministic facts (assertions, rings, repro identity ride along in additional_metadata); your judge adds semantic metrics on top. The same pattern works for LangSmith or a hand-rolled judge — verdicts are plain JSON.