Live
Testing & conformance
Three test-time tools, all serving the same captured gateway rejections:
simulate()— make the next N calls through aDonkeyreturn a chosen refusal, in process, with no server.- The conformance plugin — a pytest suite you point at your own agent to find governance bugs you did not know you had.
- The
gatewayfixture — a real simulator on an ephemeral port, for subjects that do not import the SDK, with a spy on what it received.
pip install "donkey-kit[test]" # simulate() + conformance plugin
pip install "donkey-kit[test,local]" # add the gateway fixturesimulate() — in-process, no server
A context manager that makes the next N calls through a Donkey return a
chosen refusal. It injects the same captured rejection fixture the error
classifier and the local simulator use, so your agent sees exactly
that typed refusal — not a hand-rolled stand-in. No network, no server, fast
enough for unit tests:
from donkey_kit import Donkey, PIIDetected
async def test_agent_masks_pii():
donkey = Donkey.from_env()
with donkey.simulate(PIIDetected):
result = await triage_agent.run(ticket_with_card_number)
assert "****" in result.draft_replytimes=N(default1) sets how many calls are refused before requests proceed normally. Retries of a single logical call count once.- The previous transport is restored on exit, even if the block raises, and
nested
simulate()blocks compose. - Every injected response carries
x-donkey-simulator: true, so a simulated refusal is never mistaken for a real gateway response in a log or trace. - Injected calls stay in-process even when
HTTP_PROXY,HTTPS_PROXYorALL_PROXYis set. Calls that proceed aftertimesstill go through the configured proxy.
The selectable refusals are the ones produced from a captured fixture:
TokenBudgetExceeded, RequestRateLimitExceeded, PIIDetected,
PromptInjectionBlocked, ContentSafetyBlocked, AgentKilled, UpstreamRequestError, ModelNotRoutable,
UpstreamModelError, AuthError, and the generic PolicyViolation (the
content-moderation shape). A type registered with
classify.register() is selectable too: simulate() serves
the minimal response its registration describes.
Asking for a refusal no gateway response produces — for example a client-side
ConfigError — raises a ValueError.
simulate() injects the fixture verbatim. For scripted behaviour such as
every-Nth-call PII blocks or a shrinking budget window, use the
gateway fixture with
set_scenarios(...), or donkey mock --scenario.
simulate() covers model calls only. To refuse an MCP tool call (a PII
block, schema rejection, catalog poisoning block or access denial), use
donkey.simulate_mcp("pii-request", servers=...).
Inside that block, donkey.tools.discover() returns the servers you declare,
so no gateway or Exchange is needed.
For an outbound A2A call through donkey.a2a, use
donkey.simulate_a2a("pii-send", agents=[...]).
The conformance plugin — run it against your agent
Installing donkey-kit[test] registers a pytest plugin with a conformance suite
you point at your own agent factory:
pytest --donkey-conformance --donkey-agent=my_app.agent:buildIt answers questions you probably cannot currently answer about your own code:
- Does your agent retry a
TokenBudgetExceeded? It must not — a policy refusal is terminal, and retrying it burns budget to earn the same refusal. - Does it swallow
PIIDetectedas a generic exception? - Does it propagate the correlation ID into its own logs?
- Does it still work when budget headers are absent entirely?
Each scenario becomes one pytest item, and the run prints a table of scenario → pass / fail / exempt. A failure reads as a finding about your agent — for example, “your agent retried a budget refusal 3 times” — found in CI, before production.
The suite grades what it sees on the Donkey’s transport, so your agent’s model
calls must go through the donkey it is given. A scenario in which no call
reaches that transport fails with “no model call went through the Donkey
client”, never passes by default. That covers an agent that makes no call, one
that builds its own httpx or OpenAI client, and an adapter that sends calls
outside the shared transport (CrewAI, LlamaIndex, Microsoft Agent Framework,
and ADK model()). The finding names the adapter. If the gap is expected,
record an asserted exemption for it. The run never reaches a real gateway: the
harness replaces the configured proxy URL and every credential with
placeholders, and it refuses any request that would leave the process.
In-process and server modes
--donkey-mode picks how the suite reaches your agent’s calls. The scenarios,
the verdicts and the printed table are the same in both modes.
pytest --donkey-conformance --donkey-agent=my_app.agent:build # in-process (default)
pytest --donkey-conformance --donkey-agent=my_app.agent:build --donkey-mode server # needs donkey-kit[local]in-process(the default) swaps a counting transport onto theDonkey’s HTTP client. It opens no socket, which makes it the fastest mode.serverstarts the local simulator, the same appdonkey mockserves, on an ephemeral127.0.0.1port. Each scenario’sDonkeypoints at that URL, so every call makes a real HTTP hop through your agent’s actual transport stack. Each scenario re-arms the simulator for its own refusal, and sends are counted from the server’s request log. Use this mode to check that your agent behaves the same over a socket. Requests to any other host are still refused.
Server mode replays captured gateway responses only. A scenario that needs a
type you registered with classify.register() has no
captured response, so the run stops with a usage error that points you back to
--donkey-mode in-process. Choosing server mode without the [local] extra
fails at startup with the pip install to run.
The factory is called once per scenario and receives the donkey fixture if it
declares one. With --donkey-conformance, the plugin runs the suite
exclusively in place of normal test collection; without the flag it is inert.
--agent is a deprecated alias for --donkey-agent.
You can also run it through the CLI with donkey test --agent my_app.agent:build
(see CLI).
Exemptions
Exemptions are asserted, never silently skipped. If your agent
legitimately cannot satisfy a scenario, record it in a KNOWN_LIMITATIONS
mapping of {scenario: reason} as an explicit, reviewable claim.
By default the plugin reads a KNOWN_LIMITATIONS attribute from the --donkey-agent
module; point elsewhere with --donkey-known-limitations=module:NAME. The
mapping is validated at collection time, so an unknown scenario key or an empty
reason fails the run before any scenario executes.
Custom policy registrations
classify.register() is process-wide, so a registration
made in one test would leak into the next. The plugin’s donkey fixture drops
any registration made during the test at teardown, and the
donkey_classify_registry fixture does the same while yielding the registry
itself:
def test_salary_block_is_typed(donkey_classify_registry):
donkey_classify_registry.register(
match=dict(status=403, error_type="acme_salary_block"),
raises=SalaryLeakBlocked,
)
...Registrations made at import time, by the module that defines your types, are kept.
The gateway fixture
simulate() and the conformance plugin run your subject in process. Some
subjects cannot be: a containerised agent, a Node service, an A2A client, a
manual curl. Those need a real listener on a real port — and often you need
to know what the gateway actually received. “Did my agent stop after the
refusal, or retry four more times?” is answerable only from the gateway’s side.
The gateway fixture boots the local simulator on an ephemeral
port, hands your test its .url, and records every request:
async def test_agent_stops_after_a_refusal(gateway):
gateway.set_scenarios("pii_block:every=1")
app = deploy(env={"DONKEY_LLM_PROXY_URL": gateway.url})
await app.run(ticket_with_card_number)
# A policy refusal is terminal — the agent must not retry it.
assert gateway.requests_received == 1gateway.urlis a realhttp://127.0.0.1:<port>any process can pointDONKEY_LLM_PROXY_URLat.- The port is bound to
0, so parallelpytest -nruns never collide. gateway.requests_receivedcounts requests;gateway.requestsexposes each one’s method, path, and header names. Only content-type, client/wallet IDs and reserved application/business-group attribution values are retained. Correlation IDs, call IDs, cost values and every other header value are"***", even withoutrecording_config.gateway.set_scenarios(...)arms scenario scripting (pii_block,injection,budget) per test.- Every response carries
x-donkey-simulator: true. - The server is torn down when the test exits, including on failure.
The fixture needs the [local] extra (Starlette + Uvicorn); requesting it
without that extra raises an ImportError naming the exact pip install.
You can pass your DonkeyConfig as recording_config to
gateway.configure(...) or start_gateway(...); configured
llm_proxy_key_header and user_token_header values are always redacted.
Omitting it on a later configure() retains the recorder’s current config.
The conformance harness supplies its config itself. Valid SDK configuration
reserves the retained names against custom credential use, but header names
cannot prove arbitrary values safe: non-SDK clients must not put secrets in
those names. This recorder is not a general-purpose sanitizer for arbitrary
HTTP headers.
Choosing a tool
donkey mock | gateway fixture | simulate() | |
|---|---|---|---|
| Shape | A server you run | A server a test starts | In-process context manager |
| Port | You pick it (--port) | Ephemeral (0), on .url | None |
| Use it for | Manual dev, demos, any client | Testing an out-of-process subject | Fast unit tests |
| Needs a network | Yes (localhost) | Yes (localhost) | No |
| Works with a stock OpenAI client | Yes | Yes | No — it hooks the Donkey transport |
| Asserts on what the gateway received | No | Yes (requests_received) | No |
All three read the same rejection fixtures, so they cannot drift apart.