Skip to Content
These docs describe donkey-kit v0.2.1. Release notes
Telemetry & cost

Live

Telemetry & cost

DDK answers two questions about every governed call: what happened? and who pays for it? It does so with OpenTelemetry GenAI spans, per-run correlation IDs, cost-attribution tags, and routing and resilience signals. For what the SDK itself does along the way, such as retries and token refreshes, see Debug logging.

OpenTelemetry GenAI spans

Every governed model call produces a span following the OpenTelemetry GenAI semantic conventions — gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens — plus attributes for the governance layer that generic instrumentation cannot know about. gen_ai.system is the bare vendor the gateway served (openai), even when the proxy reports a qualified mulesoft:openai; donkey.last_call.served_provider keeps the qualified value.

donkey.policy.decision = allow | refuse donkey.policy.type = pii_detected | token_budget | request_rate_limit | injection | agent_killed | approval_denied | content_safety | … donkey.policy.provider = google-model-armor # who blocked it (provider-backed refusals only) donkey.policy.categories = [rai_dangerous] # why: the provider's flagged categories donkey.budget.remaining = 18450 donkey.correlation_id = … donkey.correlation_id.source = adopted # adopted | minted, inside donkey.run_from_headers() only donkey.cost.team = support donkey.cost.project = triage-v2 donkey.cost.env = prod donkey.cost.enduser.id = user-42 donkey.routing.type = ModelBased # how the gateway routed donkey.routing.fallback = false # did it fail over? gen_ai.response.model = gpt-5.1 # the model that actually served donkey.cache.status = hit # semantic-cache outcome (on a cached proxy) donkey.cache.score = 0.9518 # similarity score, hit only donkey.usage.cached_tokens = 512 # omitted when the provider reports none donkey.usage.cache_write_tokens = 128 donkey.usage.reasoning_tokens = 96

A refusal typed by a custom policy registration records its slug verbatim as donkey.policy.type, for example acme_salary_block, so the span and your policy’s own logs share one string.

A refusal made by an external moderation provider also says who blocked the call and why. donkey.policy.provider is the provider’s stable slug (azure-content-safety, bedrock-guardrails, google-model-armor or crowdstrike-aidr, whose shape is read from the policy’s source and not captured live), and donkey.policy.categories is a string array of the categories it flagged. They are the same values as ContentSafetyBlocked.provider and .categories on the exception. Both are omitted on every other refusal, such as a PII block or a budget refusal. The categories are the provider’s fixed filter names, not request content, so they are safe to export. Group refusals by donkey.policy.provider to compare how often each provider blocks you, and by donkey.policy.categories to see which categories trip most.

The three donkey.usage.* counts carry the cost-relevant detail tokens the semantic conventions have no pinned key for — cached / cache-write prompt tokens and reasoning-model thinking tokens. They are read from the response usage block’s detail sub-objects and are omitted, never 0, when the provider reports no detail counts. When the response passes through the SDK’s shared HTTP client, the same counts are exposed per-call on donkey.last_call.

A refused request still produces a span, with donkey.policy.decision=refuse and otel.status_code=ERROR. A streaming response produces exactly one span, with token counts filled in at stream end.

DDK’s GenAI spans mark escaping exceptions as ERROR without copying exception messages, tracebacks or chained diagnostics into events or status descriptions. This holds even with content capture enabled; the original exception still propagates to the caller. Raw refusal .response bodies and the LLM PII .gateway_message remain available for explicit inspection and may contain blocked values. Host or framework instrumentation has its own recording policy.

A tool that needs approval records each step of its approval as a donkey.tool.approval span with one event (donkey.approval.pending, .approved, .rejected or .timed_out). The event names the approver and carries donkey.correlation_id, but never the tool’s arguments.

Each tool discovery is a donkey.tools.discover span, and each donkey.tools.warm() a donkey.tools.warm span. donkey.discover.cache is hit when the set came from the cache and miss when the registry was read. donkey.discover.servers counts the servers in the set, and donkey.discover.left_out the MCP assets left out as ungoverned. A failed read sets error.type to the exception’s class name, never its message.

Export goes over OTLP to wherever you already send spans. Nothing in the emit path is Anypoint-specific, so if your team already runs Grafana, Elastic, Honeycomb, New Relic, Datadog, Dynatrace, Langfuse or Phoenix, “policy refusals per hour by type” and “tokens per ticket” show up in the dashboard you already have. See Send spans to your observability backend.

Outbound A2A spans

The official A2A SDK has its own instrumentation, independent of DDK’s telemetry setting. It can record response diagnostics with an ALWAYS_ON host sampler. By default DDK refuses outbound A2A calls when SDK tracing is enabled or its state cannot be determined, before registry reads or HTTP. Disable SDK instrumentation before importing it with OTEL_INSTRUMENTATION_A2A_SDK_ENABLED=false; this does not disable DDK spans. Alternatively, explicitly set allow_a2a_sdk_telemetry=True to accept the SDK’s diagnostic-recording risk. DDK’s ordinary telemetry=True is not that permission, and telemetry=False does not disable the SDK’s recorder.

To keep both tracing systems enabled:

donkey = Donkey.from_env(allow_a2a_sdk_telemetry=True)

The equivalent configuration is DONKEY_ALLOW_A2A_SDK_TELEMETRY=true or allow_a2a_sdk_telemetry = true in the [donkey] TOML table. It defaults to false. The check uses the SDK’s imported instrumentation state: changing OTEL_INSTRUMENTATION_A2A_SDK_ENABLED after import is too late; restart the process with the opt-out set. DDK never changes the environment or the global tracer provider for you. With explicit permission, SDK spans retain their normal parentage and may export exception messages and stack traces.

donkey.a2a.send() and stream() open one donkey.a2a.call CLIENT span covering card discovery and delegation. This is not a GenAI span: A2A SSE is decoded by the official SDK and never scanned for model token usage. The shared transport still observes budget headers, last_call and typed HTTP refusals.

The span preserves the enclosing run’s correlation ID/source or temporarily mints one for the operation. It injects W3C traceparent and, when present, tracestate, without baggage. A remote DDK inbound donkey.a2a.task span can adopt that parent. Team/project/environment cost tags may be recorded, with run overrides winning; end-user IDs and raw diagnostics are not recorded on this span. No LLM cost headers are sent on the scoped A2A plane.

The stream span remains open until the async with context exits. Break, error and cancellation close local resources; cancellation does not send a remote cancel request. Failures record only error.type, with no exception message or exception event. telemetry=False disables the span and W3C propagation, but correlation still works.

This behavior is OFFLINE-VERIFIED. Successful gateway send and SSE are NOT live verified by this implementation yet; refused streaming requests before SSE are not successful streaming evidence.

The A2A policy inventory records the captured quality headers separately. Successful delegation HTTP responses record donkey.a2a.quality.evaluation="yes", a finite numeric donkey.a2a.quality.score, and donkey.a2a.quality.status="completed" when present. Missing, malformed or unknown evaluation/status values are omitted; no score range is inferred. Agent Card GETs and failed HTTP responses do not contribute quality annotations. An annotation never changes the call outcome.

donkey.a2a.quality.explanation can repeat request content. It is omitted by default and recorded only with telemetry_capture_content=True, truncated to 4096 characters. Opt-in permits sensitive content export; it does not sanitize the explanation. This recording is OFFLINE-VERIFIED against captured v0.3 headers and mock send/stream IO, not a live facade or streaming evaluation. It is new in 0.2.1.

DDK does not send X-AGENT-ID on MCP or A2A calls: its effect on mulesoft_agent_connection_count attribution has not been verified live. No connection-graph or metric guarantee is made. Auth challenge fields, returned task/context IDs and raw refusal diagnostics are not recorded on the outbound span, even with content capture on.

Zero-config export

Set the standard OpenTelemetry endpoint env var and spans flow — there is no SDK-specific variable.

pip install "donkey-kit[otel]" export OTEL_EXPORTER_OTLP_ENDPOINT=https://langfuse.acme.internal export OTEL_SERVICE_NAME=support-triage # standard OTel var, honoured for free python -m my_app # spans flow, refused calls included

Donkey.from_env() reads OTEL_EXPORTER_OTLP_ENDPOINT (or the traces-specific OTEL_EXPORTER_OTLP_TRACES_ENDPOINT) and, when one is set, sets up an OTLP exporter behind a batch processor for DDK’s own spans. The network flush runs on that background thread, off your request path, keeping instrumentation overhead under 1 ms per call. The [otel] extra ships the http/protobuf exporter; OTEL_EXPORTER_OTLP_PROTOCOL=grpc is honoured only if you also install opentelemetry-exporter-otlp-proto-grpc.

With no endpoint set, the export path is inert and silent — no exporter is built, nothing connects, nothing is printed.

DDK leaves the global provider to you

OpenTelemetry lets a process set its global TracerProvider only once. So constructing a Donkey never changes trace.get_tracer_provider() by default. The exporter above sits on a provider that only DDK’s spans use, and your other instrumentation is unaffected.

  • Your process sets its own provider, before or after Donkey() (say via opentelemetry-instrument, or a trace.set_tracer_provider(...) call during startup). DDK’s spans go to that provider from then on, through the pipeline you already set up, and DDK’s own exporter goes unused.
  • You want DDK’s provider to be the global one, so other libraries’ spans use the same exporter. Opt in:
export DONKEY_TELEMETRY_INSTALL_GLOBAL=true # or telemetry_install_global = true in .donkey-kit.toml
from donkey_kit import Donkey from donkey_kit.core.config import DonkeyConfig donkey = Donkey(DonkeyConfig.from_env().with_overrides(telemetry_install_global=True))

With the flag on and an endpoint set, the first Donkey() installs its provider as the global one, unless your process already set one. The flag does nothing without an endpoint.

DDK’s own provider flushes its pending spans when the interpreter exits. To flush earlier, call donkey.flush_telemetry(). Use it in a serverless handler before the runtime freezes the process, or in a script that checks its collector before it exits:

def handler(event, context): try: return run_agent(event) finally: donkey.flush_telemetry(timeout_s=2.0)

It flushes the provider DDK built, scoped or installed as the global by the flag, and waits up to timeout_s (default 2 seconds). It returns True once the spans are exported, or when DDK built no provider, and False if the flush timed out or failed. An export failure never raises. The call blocks, so from async code use await asyncio.to_thread(donkey.flush_telemetry).

When your process sets its own provider, DDK’s spans ride it, and flushing them is yours: trace.get_tracer_provider().force_flush(). flush_telemetry() leaves that provider alone. trace.get_tracer_provider().force_flush() does not reach DDK’s scoped provider.

One exit is flushed for you. When the gateway’s Agent Kill Switch refuses a call, DDK force-flushes both DDK’s own provider and the global one before AgentKilled reaches your code, waiting up to 2 seconds for each. So the kill’s own span, and the spans before it, are exported even if the agent exits straight away. A failed export never replaces the AgentKilled.

Opt out of telemetry entirely with a single flag:

export DONKEY_TELEMETRY=false # or telemetry = false in .donkey-kit.toml

Send spans to your observability backend

Any backend that accepts OTLP works with the same three standard variables: OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS for the vendor’s credentials, and OTEL_SERVICE_NAME. There is no DDK-specific setting and no code change between backends. Pick yours:

Grafana Cloud stores traces in Tempo. In the Grafana Cloud portal, open your stack and choose Configure on the OpenTelemetry tile. It generates the endpoint for your region and a token, already base64-encoded as instanceID:token.

export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp-gateway-prod-<region>.grafana.net/otlp" export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic%20<base64 instanceID:token>" export OTEL_SERVICE_NAME="support-triage"

Grafana’s own note for Python: write the space after Basic as %20. Find the spans in Explore → Tempo or in Application Observability. For production, Grafana recommends sending through Grafana Alloy (its OpenTelemetry Collector) rather than straight from the app.

Docs: Send data to the Grafana Cloud OTLP endpoint 

A few rules hold for every backend:

  • Base endpoint vs traces endpoint. OTEL_EXPORTER_OTLP_ENDPOINT is a base URL: the exporter appends /v1/traces. If you set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT instead, give the full URL including /v1/traces, or the backend answers 404.
  • Encode spaces in headers. Header values such as Basic <token> or ApiKey <key> should be written with %20 in place of the space; separate several headers with commas.
  • Protocol. DDK exports http/protobuf, which every backend above accepts. For gRPC, set OTEL_EXPORTER_OTLP_PROTOCOL=grpc and install opentelemetry-exporter-otlp-proto-grpc.
  • In production, consider a Collector. Sending to an OpenTelemetry Collector  (or a vendor distribution such as Grafana Alloy) lets you batch, retry, redact and fan out to several backends without touching the app.
  • Prompts stay private by default. None of these backends receives prompt or completion text unless you opt in; see Message content stays off spans by default.

Semantic-convention stability and sinks

The GenAI conventions are still Development status upstream, so attribute names can change. DDK pins the semconv version and dual-emits: gen_ai.* at the pinned version, plus a stable donkey.* namespace under this project’s control. Your dashboards do not break when upstream renames something.

DDK exports standard OTLP and leaves the sink to you. Whether Anypoint Monitoring or Agent Visualizer ingests OTLP GenAI spans is not publicly documented, so don’t rely on spans appearing there.

Message content stays off spans by default

Spans carry metadata only — model, token counts, policy decision, cost tags, correlation id. They do not carry prompt or completion text.

Spans are emitted upstream of the gateway’s PII masking. The Omni Gateway masks sensitive content in its logs; a DDK span is created inside your process, before the request reaches the gateway. Putting message text on the span would re-export the very content the platform masks — straight to whatever OTLP collector you have wired up. So capturing content is opt-in, and opting in is you assuming that obligation.

Turn it on only when your collector is a trusted sink and you have accepted responsibility for the content that lands there:

export DONKEY_TELEMETRY_CAPTURE_CONTENT=true # or: telemetry_capture_content = true under [donkey] in .donkey-kit.toml

In code, pass it to Donkey.from_env(), which takes any DonkeyConfig field as a keyword argument:

from donkey_kit import Donkey donkey = Donkey.from_env(telemetry_capture_content=True)

telemetry_capture_content resolves along the standard precedence (set in code → env → config files → default) and defaults to False. When enabled, LLM content is emitted under the pinned semconv attribute names gen_ai.prompt and gen_ai.completion. New in 0.2.1, it also permits outbound A2A quality explanations under donkey.a2a.quality.explanation, limited to 4096 characters. Gateway explanations may repeat request text and are exported without sanitization; truncation is not redaction. This A2A behavior is new in 0.2.1.

When off, prompt/completion attributes are excluded by the LLM span allowlist and A2A quality explanation text is omitted by the outbound recorder. Numeric quality scores and the captured evaluation/status values do not require content opt-in. See Outbound A2A spans.

Routing & resilience

The gateway can fail over between providers when one degrades (“Enhanced Resilience for Intelligent Routing”). It reports what it did with each request on the response — which provider and model served it, how it routed, and whether that was a fallback. DDK reads those signals off the shared transport, so you get them with no framework required — the raw donkey.llm.client() path benefits just as the deep adapters do.

Every governed call through the SDK’s shared HTTP client exposes them on donkey.last_call, beside the usage and identity fields:

donkey = Donkey.from_env() await donkey.openai().responses.create(model="gpt-5.1", input="…") r = donkey.last_call r.requested_model # "gpt-5.1" — what you asked for r.served_model # "gpt-5.1" — or a substitute after failover r.served_provider # "openai" r.routing_type # "ModelBased" — or "Semantic" on a semantic-routing proxy r.fallback # False — True if the gateway failed over r.substituted # False — served_model != requested_model r.matched_topic # None — the topic a Semantic proxy matched (else None) r.routing_score # None — that match's similarity score (else None)

On a semantic-routing proxy (routing_type == "Semantic"), the gateway also reports why it picked a provider: the topic your prompt matched and the similarity score behind that match. Those land on r.matched_topic / r.routing_score. On a model-based proxy both are None — the gateway emits no semantic header, and DDK never fabricates a value for a signal it did not observe.

They also land on the span (gen_ai.response.model, donkey.routing.type, donkey.routing.fallback, and — on a semantic route — donkey.routing.matched_topic / donkey.routing.score) — the most useful thing to have on hand when latency spikes: it tells an operator whether a slow call was routed normally or recovered from a degraded provider.

For the complete list of last_call fields — observability status, gateway identity, routing, and token usage — see the last_call field reference.

The same record also carries the call’s identity and usage: request_id (the upstream provider’s id), api_instance_id and environment_id (which gateway instance served it), and the token counts including cached and reasoning tokens. The gateway identity example walks through every field.

When last_call is unavailable

donkey.last_call is populated only when the governed response passes through the SDK’s shared HTTP client, and only in the context that made the call. CrewAI routes outside that client: its native OpenAI provider builds its own, so no response reaches the record and the run’s correlation ID isn’t sent.

LlamaIndex, Microsoft Agent Framework and ADK’s model() send through the shared client, so a call populates the record in its own context and carries the run’s correlation ID. A cold read on a Donkey that resolved only these adapters reports UNOBSERVED, and OBSERVED after a call (#740 ). ADK’s gemini() uses the shared client too (see Native Gemini for reading last_call inside an ADK run).

For a Donkey that resolved only CrewAI:

r = donkey.last_call r.status # LastCallStatus.UNAVAILABLE r.available # False r.surface # "crewai"

This is different from UNOBSERVED, which means the current context has not yet received a governed response. On an unavailable surface the SDK cannot observe any response-derived last_call field, including gateway identity, routing, fallback, and usage. If multiple non-observing adapters were resolved, surface lists their names.

An agent runner that sends the model call from a task of its own leaves its caller reading UNOBSERVED, even though the SDK observed the call: the record lands in that task. The OpenAI Agents SDK’s Runner sends a run’s first turn that way (and every turn of run_streamed), and ADK’s Runner runs the whole agent that way. Read last_call in the framework’s model hook, which runs in the call’s own task: RunHooks.on_llm_end for the OpenAI Agents SDK, after_model_callback for ADK. Calling the model directly (Strands’ Agent.invoke_async, AsyncAnthropic.messages.create, a model’s own async method) records in the caller’s context.

Two behaviours worth knowing

The SDK never double-retries a fallback. DDK retries 502/503/504 with backoff (a 502/504 on a model call only with retry_model_calls_on_gateway_errors), but if the gateway already failed over internally, a 503 it marked as a fallback is not retried again — a second recovery layer stacked on a working first one just multiplies latency against an outage the gateway already handled.

Opt in to model determinism. A silent substitution is surfaced passively on last_call.substituted by default. When a substitution is not acceptable — your evaluation, cost model and token assumptions are all pinned to one model — opt into a hard error:

donkey = Donkey.from_env(on_model_substitution="raise") # raises ModelSubstituted when the served model differs from the requested one

on_model_substitution resolves along the standard precedence (kwarg → env DONKEY_ON_MODEL_SUBSTITUTION → .donkey-kit.toml → default) and defaults to "off".

On a model-based routing proxy you address models as provider/model (for example openai/gpt-5-mini), and the gateway reports the served model without the prefix (gpt-5-mini), with the provider in its own header. A prefix that names the served provider is not counted as a difference, so asking for openai/gpt-5-mini and being served gpt-5-mini by openai is not a substitution. The same model served by a different provider still is. On a proxy with more than one provider, a bare name the gateway cannot pin to one provider is rejected with a 400 before any upstream call, and DDK raises ModelNotRoutable.

Semantic caching & semantic routing Live

Omni Gateway can answer a request from its semantic cache (a similar prompt was answered before, so no provider round-trip and no token cost) and can route semantically (pick the model by matching the prompt to a topic). Both happen at the gateway. DDK caches nothing and computes no embeddings itself; it lets you steer the gateway’s cache per request — skip it, not store a response, or override the TTL or similarity threshold via donkey.cache(...) — and reports what happened:

  • the cache outcome (hit, miss, bypass, no-store) and similarity score on donkey.last_call.cache_status / .cache_score and the span (donkey.cache.status / donkey.cache.score);
  • the matched routing topic and its score, beside the routing fields above.

A cache hit is a verbatim replay with no provider round-trip, so it never advances the token budget — the replayed usage is not fresh spend.

TypeScript parity Roadmap

The same signals will surface as donkey.lastRouting.servedModel / .fallback in the TypeScript SDK.

Correlation IDs

Set a per-run id once, and every call inside the block carries it — on the wire, on every span, and on every exception — with nothing threaded through your framework state.

async with donkey.run(id=ticket.id): await triage_agent.run(ticket)

donkey.run(id=…) binds two ids:

  • a run id → the X-Correlation-Id request header → the span’s donkey.correlation_id → DonkeyError.correlation_id. Shared by every call in the block, so a client-side log line joins to the gateway’s own record.
  • a fresh per-call id → the X-Donkey-Request-Id request header → DonkeyError.call_id. Unique per logical request, stable across that request’s retries, so one call is pinpointable within a run.

The gateway reads the inbound X-Correlation-Id and echoes it verbatim on the response, so the request and response x-correlation-id are the same value. X-Donkey-Request-Id is a client-owned per-call id the gateway does not consume. Both header names are overridable (correlation_header / call_id_header) for a gateway that expects different ones.

Propagation is contextvar-based, so it reaches through framework nodes (every LangGraph node, for instance) without threading an argument through every function, and concurrent runs never leak into each other. It works with or without OpenTelemetry installed. donkey.run(...) is a dual sync/async context manager (plain with works too); nested blocks rebind then restore.

Concurrent fan-out. Every call in a fan-out started inside donkey.run() (asyncio.gather, parallel nodes, tool calls on child tasks) carries exactly one correlation id. Each task copies the context when it is created, so this holds even for a task that outlives the block, and a task that opens its own donkey.run() changes only its own id, never its parent’s or a sibling’s. Threads do not inherit the context: run them under contextvars.copy_context() or open a donkey.run() inside the thread. Outside a run nothing is grouped. Each call gets its own fresh id, so N concurrent calls send N ids.

A refusal leaving the block is re-raised as its typed DonkeyError, whichever framework wrapped it on the way out, so except PIIDetected works the same around every framework. See Typed refusals at the framework boundary; donkey.run(typed_refusals=False) turns it off.

Across agent hops

An agent that serves requests can join its caller’s run instead of starting a fresh one. donkey.run_from_headers() reads the configured correlation header from the inbound request, adopts the caller’s id when it is well-formed ([A-Za-z0-9._:-], at most 128 characters), and mints a fresh one otherwise. A malformed id is never forwarded. Every governed call in the block then carries it, exactly as inside donkey.run(id=…).

@app.post("/tasks") async def handle(request: Request, task: Task): async with donkey.run_from_headers(request.headers): return await agent.run(task)

For an outbound hop the governed transport does not carry, such as another agent’s endpoint or an MCP server reached on your own HTTP client, pass donkey.correlation_headers(). It returns the bound run id under the configured header, or {} outside a run, and never the gateway credentials.

async with donkey.run(id=ticket.id): await agents.post(sub_agent_url, json=task, headers=donkey.correlation_headers())

The sub-agent’s spans and the gateway’s record then share one id end to end. The GenAI span inside run_from_headers() records donkey.correlation_id.source: adopted when the caller supplied the id, minted when it did not. A served A2A task applies the same rule on its own.

Run ledger

donkey.last_call keeps only the most recent call. A donkey.run() block also keeps a run ledger: every governed model call made in the run, every refusal the gateway returned, and the run’s token totals. Read it inside the block at donkey.current_run. Keep the scope to read it after the block ends:

from donkey_kit import PIIDetected scope = donkey.run(id=ticket.id) async with scope as run_id: # still the run's id, a str await triage_agent.run(ticket) run = donkey.current_run if any(r.is_a(PIIDetected) for r in run.refusals): ... # redact the later prompts ledger = scope.ledger # still readable after the block ledger.call_count # every governed model call in the run ledger.usage.total_tokens # summed across them, cached and reasoning tokens too [c.served_model for c in ledger.calls] # each entry is that call's last_call record
FieldWhat it holds
callsEach call’s last_call record, oldest first: served model and provider, fallback, request id, usage, semantic-cache and semantic-routing outcome
refusalsA RunRefusal for each policy refusal (PIIDetected, TokenBudgetExceeded, …) the gateway returned on a call in the run, even when a framework caught it, and for any ModelSubstituted the SDK raised: its kind, policy, status_code, the three ids and, for PII, the entity types. Test it with r.is_a(PIIDetected)
usageA RunUsage with the run’s input_tokens, output_tokens, total_tokens, cached_tokens, cache_write_tokens and reasoning_tokens, and calls_with_usage
call_count, refusal_countTotals for the whole run
droppedCalls and refusals the cap dropped from calls and refusals
available, surfaceFalse, with the adapter names, when nothing used can feed the ledger

Each read returns a frozen snapshot, so read again for newer state. The ledger holds metadata only, never prompt or completion text. A refusal is recorded as a RunRefusal, not the exception, because the exception’s .response keeps the gateway’s body, which can echo the values a policy blocked.

  • Scope. The ledger follows the run’s correlation id. Framework-spawned tasks append to their run’s ledger, concurrent runs never see each other’s, and a nested donkey.run() keeps its own and restores the outer one on exit. Appends are thread-safe for threads run under contextvars.copy_context().
  • Bounded memory. calls and refusals each keep the newest 1000 entries (donkey.run(ledger_max_entries=…)). Older ones are counted in dropped. The totals keep counting past the cap, so a long run’s cost stays exact.
  • Same reach as last_call. Only responses through the SDK’s shared HTTP client reach it. When every adapter used routes outside that client (CrewAI today), available is False and surface names them: the ledger is unavailable, not empty. See When last_call is unavailable.

The decorator form

When a whole function should be one run, @donkey.governed is the decorator equivalent of wrapping its body in donkey.run():

@donkey.governed(team="support") async def handle_ticket(ticket): await triage_agent.run(ticket)

Each call opens its own run — a fresh run/correlation id — and binds the optional per-run cost tags, the OTel span, and typed refusals, exactly the scope donkey.run() establishes. It wraps both sync and async callables and is usable bare (@donkey.governed) or parametrised. There is deliberately no id=: pinning one id across every call would collapse unrelated runs into a single correlation, so when you need a specific id, use donkey.run(id=…) directly.

Cost-attribution tags

A small, fixed set of tags (team, project, env, enduser.id) set once and recorded on every model call’s span:

donkey = Donkey.from_env(team="support", project="triage-v2", env="prod") async with donkey.run(id=ticket.id, enduser_id=agent_user.id): await triage_agent.run(ticket)

The tags resolve in this order: Donkey.from_env(team=…) kwargs, then DONKEY_COST_* env vars, then a [donkey.cost] table in the config files (see Configuration). Per-run overrides layer on top: donkey.run(team=…, project=…, env=…, enduser_id=…) wins per field for its block, and the rest fall back to the configured tags. The key set is fixed: an unknown dimension is a configuration error, never a silently dropped tag. Values are validated (fixed keys, bounded length), so nobody stuffs a JSON blob into a span attribute or header.

Where the tags go

CarrierNamesWhenGoes to
Span attributesdonkey.cost.team, donkey.cost.project, donkey.cost.env, donkey.cost.enduser.idAlways, for each tag that is set, on every donkey.llm.chat span. Needs telemetry on (the default) and OpenTelemetry installed. The SDK’s shared client opens the span, so CrewAI, which has only header injection, records none.Your OpenTelemetry pipeline: the OTLP exporter when OTEL_EXPORTER_OTLP_ENDPOINT is set, or any TracerProvider your process installed
Request headersX-Anypoint-Cost-Team, X-Anypoint-Cost-Project, X-Anypoint-Cost-Env, X-Anypoint-Cost-Enduser-Id (renamable with cost_*_header)Only with send_cost_headers / DONKEY_SEND_COST_HEADERS=true. Off by default.Model requests to the LLM proxy only. See the note below.

Each header carries the tag’s value unchanged (for example X-Anypoint-Cost-Enduser-Id: user-42). A tag that isn’t set is sent under neither carrier.

The Anypoint LLM Gateway doesn’t read cost tags from request headers: it meters cost from token usage per API instance and consuming client application. So the span attributes are the carrier that matters. Turn the headers on only when something of your own reads them.

With send_cost_headers on, the headers go only on model requests to the LLM proxy. The connected-app token request and other Anypoint control-plane requests never carry them, so the end-user ID doesn’t reach those hosts. Adapters with transport injection also send per-run donkey.run(...) values; adapters that get only a default_headers snapshot send the configured tags.

The end-user ID reaches your trace backend regardless of send_cost_headers. If enduser_id is set, every model-call span carries donkey.cost.enduser.id, and any exporter you’ve configured receives it. To keep it out of traces, don’t set enduser_id (DONKEY_COST_ENDUSER_ID, "enduser.id" in [donkey.cost], or donkey.run(enduser_id=…)), drop the attribute in your collector, or turn spans off with DONKEY_TELEMETRY=false.

Per-request attribution

One process often serves several tenants or teams: a shared agent platform, a multi-tenant SaaS agent, or one job that spans two cost centres. Keep one Donkey (one client, pool, auth cache and budget) and say per request which group a call belongs to:

async def handle(tenant: str, prompt: str): with donkey.attribution(business_group=tenant): return await model.ainvoke(prompt)

donkey.attribution(business_group=…, team=…, project=…, env=…, enduser_id=…) attributes every governed call in its block, on both the async and the blocking client. The business group replaces the configured business_group for the block. The precedence is this scope, then the Donkey config, then DONKEY_BUSINESS_GROUP, then the config file. The cost tags merge per field over any already bound by an enclosing block.

It doesn’t start a run, so the correlation ID stays the same and one run can span two cost centres:

async with donkey.run(id=job.id): with donkey.attribution(business_group="risk-eu"): await score(job) with donkey.attribution(business_group="risk-us"): await score(job)

donkey.run(business_group=…) and @donkey.governed(business_group=…) take the same argument when the group belongs to the whole run. Like the run ID, the binding is contextvar-based: it reaches framework-spawned tasks, concurrent tasks never see each other’s group, and nested blocks restore the outer value on exit. The value is validated like a cost tag, and an invalid one raises ConfigError where you set it.

Adapters that get only a default_headers snapshot don’t follow the scope. They captured the configured business group when they were built, so they keep sending it. Adapters with transport injection send the scoped value. The header name, X-Anypoint-Business-Group, is still an unverified placeholder; see Configuration.

The question this answers

Finance asks what the support agent cost last month versus the HR bot. Without tags, both agents share one client_id and there is no way to split the bill. With tags it is a group-by.

For compliance — “prove the HR bot’s answer to user X on date Y went through the content-safety policy” — the correlation ID on the log line joins to the gateway record, and the span carries donkey.cost.enduser.id and donkey.policy.type. An EU AI Act Article 12 log request becomes one query, not an investigation.

Debug logging

Spans tell you what a call did. When you need to see what the SDK did for you along the way, such as a retry, a token refresh or a fallback, turn on its DEBUG logging. It uses the standard library logging module, with one logger per module under the donkey_kit package logger:

import logging logging.basicConfig() # or your own handlers logging.getLogger("donkey_kit").setLevel(logging.DEBUG)

A retried 503 then reads:

DEBUG:donkey_kit.core.transport.observe:GET https://proxy.example/v1/chat/completions returned 503; retry 1 of 3 in 0.48s

The SDK attaches only a NullHandler to donkey_kit and never configures a handler, level or format of its own, so nothing prints until your application does. That also means you can narrow it. Set the level on one module logger, such as donkey_kit.core.auth, instead of the package.

What is logged at DEBUGLogger
Each retry: the status, which retry it is, and the delaydonkey_kit.core.transport.observe
A retryable status that is not retried: after a gateway routing fallback, a 502/504 on a model call (see Retry behaviour), or once retries run outdonkey_kit.core.transport.observe
A 401 on the async client, followed by one token refresh and re-senddonkey_kit.core.transport.async_client
A connected-app token fetch (the token URL only) and token invalidationdonkey_kit.core.auth
An auth provider in a ChainedAuth failing over to the next one (type names only)donkey_kit.core.auth
The OTLP export bootstrap: inert, kept your TracerProvider, or installed (global or DDK-scoped)donkey_kit.core.telemetry
How many tool descriptors a ToolSet exposes after its filter, out of the totaldonkey_kit.tools.session
A rate-limit reset header that was ignored as negative or out of range, naming the budget window (never the value)donkey_kit.core._window
A refusal translator that failed and was skipped, or a refusal capture that could not tag a framework error (type names only, no hook representation or traceback)donkey_kit.core.refusals
Other best-effort steps that failed and were skipped (with the traceback), e.g. recording usage on a spanthe module that skipped it

A log record never carries a credential. Records name a request by method, scheme, host and path only. They never include a header value, token, client secret, query string or request body, and an auth failure is logged by exception type, not message. A unit test pins this against every credential the SDK sends: the bearer token, client id and client secret.

If every ChainedAuth provider fails, the final AuthError names only the last exception type, not its message, and suppresses exception chaining. Refusal-handler failures likewise omit callback messages and tracebacks; the donkey_kit.core.reactions logger records only the handler label and exception type at WARNING, and the original refusal still propagates.

The SDK does not use WARNING log records for configuration problems. A one-time problem you have to fix is a Python warning instead, raised through warnings.warn with its own category, so you can filter it or make it an error with the warnings module. For example, an OTLP endpoint set without its exporter installed raises donkey_kit.core.telemetry.TelemetryExportWarning (see Zero-config export).

Last updated on