Skip to Content
These docs describe donkey-kit v0.2.1. Release notes
Testing & conformance

Live

Testing & conformance

Three test-time tools, all serving the same captured gateway rejections:

  • simulate() — make the next N calls through a Donkey return a chosen refusal, in process, with no server.
  • The conformance plugin — a pytest suite you point at your own agent to find governance bugs you did not know you had.
  • The gateway fixture — a real simulator on an ephemeral port, for subjects that do not import the SDK, with a spy on what it received.
pip install "donkey-kit[test]" # simulate() + conformance plugin pip install "donkey-kit[test,local]" # add the gateway fixture

simulate() — in-process, no server

A context manager that makes the next N calls through a Donkey return a chosen refusal. It injects the same captured rejection fixture the error classifier and the local simulator use, so your agent sees exactly that typed refusal — not a hand-rolled stand-in. No network, no server, fast enough for unit tests:

from donkey_kit import Donkey, PIIDetected async def test_agent_masks_pii(): donkey = Donkey.from_env() with donkey.simulate(PIIDetected): result = await triage_agent.run(ticket_with_card_number) assert "****" in result.draft_reply
  • times=N (default 1) sets how many calls are refused before requests proceed normally. Retries of a single logical call count once.
  • The previous transport is restored on exit, even if the block raises, and nested simulate() blocks compose.
  • Every injected response carries x-donkey-simulator: true, so a simulated refusal is never mistaken for a real gateway response in a log or trace.
  • Injected calls stay in-process even when HTTP_PROXY, HTTPS_PROXY or ALL_PROXY is set. Calls that proceed after times still go through the configured proxy.

The selectable refusals are the ones produced from a captured fixture: TokenBudgetExceeded, RequestRateLimitExceeded, PIIDetected, PromptInjectionBlocked, ContentSafetyBlocked, AgentKilled, UpstreamRequestError, ModelNotRoutable, UpstreamModelError, AuthError, and the generic PolicyViolation (the content-moderation shape). A type registered with classify.register() is selectable too: simulate() serves the minimal response its registration describes. Asking for a refusal no gateway response produces — for example a client-side ConfigError — raises a ValueError.

simulate() injects the fixture verbatim. For scripted behaviour such as every-Nth-call PII blocks or a shrinking budget window, use the gateway fixture with set_scenarios(...), or donkey mock --scenario.

simulate() covers model calls only. To refuse an MCP tool call (a PII block, schema rejection, catalog poisoning block or access denial), use donkey.simulate_mcp("pii-request", servers=...). Inside that block, donkey.tools.discover() returns the servers you declare, so no gateway or Exchange is needed. For an outbound A2A call through donkey.a2a, use donkey.simulate_a2a("pii-send", agents=[...]).

The conformance plugin — run it against your agent

Installing donkey-kit[test] registers a pytest plugin with a conformance suite you point at your own agent factory:

pytest --donkey-conformance --donkey-agent=my_app.agent:build

It answers questions you probably cannot currently answer about your own code:

  • Does your agent retry a TokenBudgetExceeded? It must not — a policy refusal is terminal, and retrying it burns budget to earn the same refusal.
  • Does it swallow PIIDetected as a generic exception?
  • Does it propagate the correlation ID into its own logs?
  • Does it still work when budget headers are absent entirely?

Each scenario becomes one pytest item, and the run prints a table of scenario → pass / fail / exempt. A failure reads as a finding about your agent — for example, “your agent retried a budget refusal 3 times” — found in CI, before production.

The suite grades what it sees on the Donkey’s transport, so your agent’s model calls must go through the donkey it is given. A scenario in which no call reaches that transport fails with “no model call went through the Donkey client”, never passes by default. That covers an agent that makes no call, one that builds its own httpx or OpenAI client, and an adapter that sends calls outside the shared transport (CrewAI, LlamaIndex, Microsoft Agent Framework, and ADK model()). The finding names the adapter. If the gap is expected, record an asserted exemption for it. The run never reaches a real gateway: the harness replaces the configured proxy URL and every credential with placeholders, and it refuses any request that would leave the process.

In-process and server modes

--donkey-mode picks how the suite reaches your agent’s calls. The scenarios, the verdicts and the printed table are the same in both modes.

pytest --donkey-conformance --donkey-agent=my_app.agent:build # in-process (default) pytest --donkey-conformance --donkey-agent=my_app.agent:build --donkey-mode server # needs donkey-kit[local]
  • in-process (the default) swaps a counting transport onto the Donkey’s HTTP client. It opens no socket, which makes it the fastest mode.
  • server starts the local simulator, the same app donkey mock serves, on an ephemeral 127.0.0.1 port. Each scenario’s Donkey points at that URL, so every call makes a real HTTP hop through your agent’s actual transport stack. Each scenario re-arms the simulator for its own refusal, and sends are counted from the server’s request log. Use this mode to check that your agent behaves the same over a socket. Requests to any other host are still refused.

Server mode replays captured gateway responses only. A scenario that needs a type you registered with classify.register() has no captured response, so the run stops with a usage error that points you back to --donkey-mode in-process. Choosing server mode without the [local] extra fails at startup with the pip install to run.

The factory is called once per scenario and receives the donkey fixture if it declares one. With --donkey-conformance, the plugin runs the suite exclusively in place of normal test collection; without the flag it is inert. --agent is a deprecated alias for --donkey-agent. You can also run it through the CLI with donkey test --agent my_app.agent:build (see CLI).

Exemptions

Exemptions are asserted, never silently skipped. If your agent legitimately cannot satisfy a scenario, record it in a KNOWN_LIMITATIONS mapping of {scenario: reason} as an explicit, reviewable claim.

By default the plugin reads a KNOWN_LIMITATIONS attribute from the --donkey-agent module; point elsewhere with --donkey-known-limitations=module:NAME. The mapping is validated at collection time, so an unknown scenario key or an empty reason fails the run before any scenario executes.

Custom policy registrations

classify.register() is process-wide, so a registration made in one test would leak into the next. The plugin’s donkey fixture drops any registration made during the test at teardown, and the donkey_classify_registry fixture does the same while yielding the registry itself:

def test_salary_block_is_typed(donkey_classify_registry): donkey_classify_registry.register( match=dict(status=403, error_type="acme_salary_block"), raises=SalaryLeakBlocked, ) ...

Registrations made at import time, by the module that defines your types, are kept.

The gateway fixture

simulate() and the conformance plugin run your subject in process. Some subjects cannot be: a containerised agent, a Node service, an A2A client, a manual curl. Those need a real listener on a real port — and often you need to know what the gateway actually received. “Did my agent stop after the refusal, or retry four more times?” is answerable only from the gateway’s side.

The gateway fixture boots the local simulator on an ephemeral port, hands your test its .url, and records every request:

async def test_agent_stops_after_a_refusal(gateway): gateway.set_scenarios("pii_block:every=1") app = deploy(env={"DONKEY_LLM_PROXY_URL": gateway.url}) await app.run(ticket_with_card_number) # A policy refusal is terminal — the agent must not retry it. assert gateway.requests_received == 1
  • gateway.url is a real http://127.0.0.1:<port> any process can point DONKEY_LLM_PROXY_URL at.
  • The port is bound to 0, so parallel pytest -n runs never collide.
  • gateway.requests_received counts requests; gateway.requests exposes each one’s method, path, and header names. Only content-type, client/wallet IDs and reserved application/business-group attribution values are retained. Correlation IDs, call IDs, cost values and every other header value are "***", even without recording_config.
  • gateway.set_scenarios(...) arms scenario scripting (pii_block, injection, budget) per test.
  • Every response carries x-donkey-simulator: true.
  • The server is torn down when the test exits, including on failure.

The fixture needs the [local] extra (Starlette + Uvicorn); requesting it without that extra raises an ImportError naming the exact pip install.

You can pass your DonkeyConfig as recording_config to gateway.configure(...) or start_gateway(...); configured llm_proxy_key_header and user_token_header values are always redacted. Omitting it on a later configure() retains the recorder’s current config. The conformance harness supplies its config itself. Valid SDK configuration reserves the retained names against custom credential use, but header names cannot prove arbitrary values safe: non-SDK clients must not put secrets in those names. This recorder is not a general-purpose sanitizer for arbitrary HTTP headers.

Choosing a tool

donkey mockgateway fixturesimulate()
ShapeA server you runA server a test startsIn-process context manager
PortYou pick it (--port)Ephemeral (0), on .urlNone
Use it forManual dev, demos, any clientTesting an out-of-process subjectFast unit tests
Needs a networkYes (localhost)Yes (localhost)No
Works with a stock OpenAI clientYesYesNo — it hooks the Donkey transport
Asserts on what the gateway receivedNoYes (requests_received)No

All three read the same rejection fixtures, so they cannot drift apart.

Last updated on