Live
Telemetry & cost
DDK answers two questions about every governed call: what happened? and who pays for it? It does so with OpenTelemetry GenAI spans, per-run correlation IDs, cost-attribution tags, and routing and resilience signals. For what the SDK itself does along the way, such as retries and token refreshes, see Debug logging.
OpenTelemetry GenAI spans
Every governed model call produces a span following the OpenTelemetry GenAI
semantic conventions — gen_ai.system, gen_ai.request.model,
gen_ai.usage.input_tokens, gen_ai.usage.output_tokens — plus attributes
for the governance layer that generic instrumentation cannot know about.
gen_ai.system is the bare vendor the gateway served (openai), even when the
proxy reports a qualified mulesoft:openai; donkey.last_call.served_provider
keeps the qualified value.
donkey.policy.decision = allow | refuse
donkey.policy.type = pii_detected | token_budget | request_rate_limit | injection | agent_killed | approval_denied | content_safety | …
donkey.policy.provider = google-model-armor # who blocked it (provider-backed refusals only)
donkey.policy.categories = [rai_dangerous] # why: the provider's flagged categories
donkey.budget.remaining = 18450
donkey.correlation_id = …
donkey.correlation_id.source = adopted # adopted | minted, inside donkey.run_from_headers() only
donkey.cost.team = support
donkey.cost.project = triage-v2
donkey.cost.env = prod
donkey.cost.enduser.id = user-42
donkey.routing.type = ModelBased # how the gateway routed
donkey.routing.fallback = false # did it fail over?
gen_ai.response.model = gpt-5.1 # the model that actually served
donkey.cache.status = hit # semantic-cache outcome (on a cached proxy)
donkey.cache.score = 0.9518 # similarity score, hit only
donkey.usage.cached_tokens = 512 # omitted when the provider reports none
donkey.usage.cache_write_tokens = 128
donkey.usage.reasoning_tokens = 96A refusal typed by a custom policy registration records its
slug verbatim as donkey.policy.type, for example acme_salary_block, so the
span and your policy’s own logs share one string.
A refusal made by an external moderation provider also says who blocked the
call and why. donkey.policy.provider is the provider’s stable slug
(azure-content-safety, bedrock-guardrails, google-model-armor or
crowdstrike-aidr, whose shape is read from the policy’s source and not
captured live), and donkey.policy.categories is a string array of the
categories it flagged. They are the same values as ContentSafetyBlocked.provider
and .categories on the exception. Both are omitted on every other
refusal, such as a PII block or a budget refusal. The categories are the
provider’s fixed filter names, not request content, so they are safe to export.
Group refusals by donkey.policy.provider to compare how often each provider
blocks you, and by donkey.policy.categories to see which categories trip most.
The three donkey.usage.* counts carry the cost-relevant detail tokens the
semantic conventions have no pinned key for — cached / cache-write prompt tokens
and reasoning-model thinking tokens. They are read from the response usage
block’s detail sub-objects and are omitted, never 0, when the provider
reports no detail counts. When the response passes through the SDK’s shared HTTP
client, the same counts are exposed per-call on donkey.last_call.
A refused request still produces a span, with donkey.policy.decision=refuse
and otel.status_code=ERROR. A streaming response produces exactly one span,
with token counts filled in at stream end.
DDK’s GenAI spans mark escaping exceptions as ERROR without copying exception
messages, tracebacks or chained diagnostics into events or status descriptions.
This holds even with content capture enabled; the original exception still
propagates to the caller. Raw refusal .response bodies and the LLM PII
.gateway_message remain available for explicit inspection and may contain
blocked values. Host or framework instrumentation has its own recording policy.
A tool that needs approval records each step of its approval as a
donkey.tool.approval span with one event (donkey.approval.pending,
.approved, .rejected or .timed_out). The event names the approver and
carries donkey.correlation_id, but never the tool’s arguments.
Each tool discovery is a donkey.tools.discover span,
and each donkey.tools.warm() a donkey.tools.warm span. donkey.discover.cache
is hit when the set came from the cache and miss when the registry was read.
donkey.discover.servers counts the servers in the set, and
donkey.discover.left_out the MCP assets left out as ungoverned. A failed read
sets error.type to the exception’s class name, never its message.
Export goes over OTLP to wherever you already send spans. Nothing in the emit path is Anypoint-specific, so if your team already runs Grafana, Elastic, Honeycomb, New Relic, Datadog, Dynatrace, Langfuse or Phoenix, “policy refusals per hour by type” and “tokens per ticket” show up in the dashboard you already have. See Send spans to your observability backend.
Outbound A2A spans
The official A2A SDK has its own instrumentation, independent of DDK’s
telemetry setting. It can record response diagnostics with an ALWAYS_ON
host sampler. By default DDK refuses outbound A2A calls when SDK tracing is
enabled or its state cannot be determined, before registry reads or HTTP.
Disable SDK instrumentation before importing it with
OTEL_INSTRUMENTATION_A2A_SDK_ENABLED=false; this does not disable DDK spans.
Alternatively, explicitly set allow_a2a_sdk_telemetry=True to accept the
SDK’s diagnostic-recording risk. DDK’s ordinary telemetry=True is not that
permission, and telemetry=False does not disable the SDK’s recorder.
To keep both tracing systems enabled:
donkey = Donkey.from_env(allow_a2a_sdk_telemetry=True)The equivalent configuration is DONKEY_ALLOW_A2A_SDK_TELEMETRY=true or
allow_a2a_sdk_telemetry = true in the [donkey] TOML table. It defaults to
false. The check uses the SDK’s imported instrumentation state: changing
OTEL_INSTRUMENTATION_A2A_SDK_ENABLED after import is too late; restart the
process with the opt-out set. DDK never changes the environment or the global
tracer provider for you. With explicit permission, SDK spans retain their
normal parentage and may export exception messages and stack traces.
donkey.a2a.send() and stream() open one
donkey.a2a.call CLIENT span covering card discovery and delegation. This is
not a GenAI span: A2A SSE is decoded by the official SDK and never scanned for
model token usage. The shared transport still observes budget headers,
last_call and typed HTTP refusals.
The span preserves the enclosing run’s correlation ID/source or temporarily
mints one for the operation. It injects W3C traceparent and, when present,
tracestate, without baggage. A remote DDK inbound donkey.a2a.task span can
adopt that parent. Team/project/environment cost tags may be recorded, with
run overrides winning; end-user IDs and raw diagnostics are not recorded on
this span. No LLM cost headers are
sent on the scoped A2A plane.
The stream span remains open until the async with context exits. Break,
error and cancellation close local resources; cancellation does not send a
remote cancel request. Failures record only error.type, with no exception
message or exception event. telemetry=False disables the span and W3C
propagation, but correlation still works.
This behavior is OFFLINE-VERIFIED. Successful gateway send and SSE are NOT live verified by this implementation yet; refused streaming requests before SSE are not successful streaming evidence.
The A2A policy inventory records
the captured quality headers separately. Successful delegation HTTP responses
record donkey.a2a.quality.evaluation="yes", a finite numeric
donkey.a2a.quality.score, and donkey.a2a.quality.status="completed" when
present. Missing, malformed or unknown evaluation/status values are omitted;
no score range is inferred. Agent Card GETs and failed HTTP responses do not
contribute quality annotations. An annotation never changes the call outcome.
donkey.a2a.quality.explanation can repeat request content. It is omitted by
default and recorded only with telemetry_capture_content=True, truncated to
4096 characters. Opt-in permits sensitive content export; it does not sanitize
the explanation. This recording is OFFLINE-VERIFIED against captured v0.3
headers and mock send/stream IO, not a live facade or streaming evaluation.
It is new in 0.2.1.
DDK does not send X-AGENT-ID on MCP or A2A calls: its effect on
mulesoft_agent_connection_count attribution has not been verified live.
No connection-graph or metric guarantee is made. Auth challenge fields, returned task/context IDs and raw refusal
diagnostics are not recorded on the outbound span, even with content capture on.
Zero-config export
Set the standard OpenTelemetry endpoint env var and spans flow — there is no SDK-specific variable.
pip install "donkey-kit[otel]"
export OTEL_EXPORTER_OTLP_ENDPOINT=https://langfuse.acme.internal
export OTEL_SERVICE_NAME=support-triage # standard OTel var, honoured for free
python -m my_app # spans flow, refused calls includedDonkey.from_env() reads OTEL_EXPORTER_OTLP_ENDPOINT (or the traces-specific
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT) and, when one is set, sets up an OTLP
exporter behind a batch processor for DDK’s own spans. The network flush runs on that background
thread, off your request path, keeping instrumentation overhead under
1 ms per call. The [otel] extra ships the http/protobuf exporter;
OTEL_EXPORTER_OTLP_PROTOCOL=grpc is honoured only if you also install
opentelemetry-exporter-otlp-proto-grpc.
With no endpoint set, the export path is inert and silent — no exporter is built, nothing connects, nothing is printed.
DDK leaves the global provider to you
OpenTelemetry lets a process set its global TracerProvider only once. So
constructing a Donkey never changes trace.get_tracer_provider() by
default. The exporter above sits on a provider that only DDK’s spans use, and
your other instrumentation is unaffected.
- Your process sets its own provider, before or after
Donkey()(say viaopentelemetry-instrument, or atrace.set_tracer_provider(...)call during startup). DDK’s spans go to that provider from then on, through the pipeline you already set up, and DDK’s own exporter goes unused. - You want DDK’s provider to be the global one, so other libraries’ spans use the same exporter. Opt in:
export DONKEY_TELEMETRY_INSTALL_GLOBAL=true # or telemetry_install_global = true in .donkey-kit.tomlfrom donkey_kit import Donkey
from donkey_kit.core.config import DonkeyConfig
donkey = Donkey(DonkeyConfig.from_env().with_overrides(telemetry_install_global=True))With the flag on and an endpoint set, the first Donkey() installs its provider
as the global one, unless your process already set one. The flag does nothing
without an endpoint.
DDK’s own provider flushes its pending spans when the interpreter exits. To
flush earlier, call donkey.flush_telemetry(). Use it in a serverless handler
before the runtime freezes the process, or in a script that checks its collector
before it exits:
def handler(event, context):
try:
return run_agent(event)
finally:
donkey.flush_telemetry(timeout_s=2.0)It flushes the provider DDK built, scoped or installed as the global by the
flag, and waits up to timeout_s (default 2 seconds). It returns True once
the spans are exported, or when DDK built no provider, and False if the flush
timed out or failed. An export failure never raises. The call blocks, so from
async code use await asyncio.to_thread(donkey.flush_telemetry).
When your process sets its own provider, DDK’s spans ride it, and flushing them
is yours: trace.get_tracer_provider().force_flush(). flush_telemetry()
leaves that provider alone. trace.get_tracer_provider().force_flush() does not
reach DDK’s scoped provider.
One exit is flushed for you. When the gateway’s Agent Kill Switch refuses a call,
DDK force-flushes both DDK’s own provider and the global one before
AgentKilled reaches your code, waiting up to 2 seconds for each. So
the kill’s own span, and the spans before it, are exported even if the agent exits
straight away. A failed export never replaces the AgentKilled.
Opt out of telemetry entirely with a single flag:
export DONKEY_TELEMETRY=false # or telemetry = false in .donkey-kit.tomlSend spans to your observability backend
Any backend that accepts OTLP works with the same three standard variables:
OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS for the vendor’s
credentials, and OTEL_SERVICE_NAME. There is no DDK-specific setting and no
code change between backends. Pick yours:
Grafana Cloud
Grafana Cloud stores traces in Tempo. In the Grafana Cloud portal, open your
stack and choose Configure on the OpenTelemetry tile. It generates the
endpoint for your region and a token, already base64-encoded as
instanceID:token.
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp-gateway-prod-<region>.grafana.net/otlp"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic%20<base64 instanceID:token>"
export OTEL_SERVICE_NAME="support-triage"Grafana’s own note for Python: write the space after Basic as %20. Find the
spans in Explore → Tempo or in Application Observability. For
production, Grafana recommends sending through Grafana Alloy (its
OpenTelemetry Collector) rather than straight from the app.
A few rules hold for every backend:
- Base endpoint vs traces endpoint.
OTEL_EXPORTER_OTLP_ENDPOINTis a base URL: the exporter appends/v1/traces. If you setOTEL_EXPORTER_OTLP_TRACES_ENDPOINTinstead, give the full URL including/v1/traces, or the backend answers404. - Encode spaces in headers. Header values such as
Basic <token>orApiKey <key>should be written with%20in place of the space; separate several headers with commas. - Protocol. DDK exports
http/protobuf, which every backend above accepts. For gRPC, setOTEL_EXPORTER_OTLP_PROTOCOL=grpcand installopentelemetry-exporter-otlp-proto-grpc. - In production, consider a Collector. Sending to an OpenTelemetry Collector (or a vendor distribution such as Grafana Alloy) lets you batch, retry, redact and fan out to several backends without touching the app.
- Prompts stay private by default. None of these backends receives prompt or completion text unless you opt in; see Message content stays off spans by default.
Semantic-convention stability and sinks
The GenAI conventions are still Development status upstream, so
attribute names can change. DDK pins the semconv version and
dual-emits: gen_ai.* at the pinned version, plus a stable donkey.*
namespace under this project’s control. Your dashboards do not break when
upstream renames something.
DDK exports standard OTLP and leaves the sink to you. Whether Anypoint Monitoring or Agent Visualizer ingests OTLP GenAI spans is not publicly documented, so don’t rely on spans appearing there.
Message content stays off spans by default
Spans carry metadata only — model, token counts, policy decision, cost tags, correlation id. They do not carry prompt or completion text.
Spans are emitted upstream of the gateway’s PII masking. The Omni Gateway masks sensitive content in its logs; a DDK span is created inside your process, before the request reaches the gateway. Putting message text on the span would re-export the very content the platform masks — straight to whatever OTLP collector you have wired up. So capturing content is opt-in, and opting in is you assuming that obligation.
Turn it on only when your collector is a trusted sink and you have accepted responsibility for the content that lands there:
export DONKEY_TELEMETRY_CAPTURE_CONTENT=true
# or: telemetry_capture_content = true under [donkey] in .donkey-kit.tomlIn code, pass it to Donkey.from_env(), which takes any DonkeyConfig field as
a keyword argument:
from donkey_kit import Donkey
donkey = Donkey.from_env(telemetry_capture_content=True)telemetry_capture_content resolves along the standard
precedence (set in code → env → config
files → default) and defaults to False. When
enabled, LLM content is emitted under the pinned semconv attribute names
gen_ai.prompt and gen_ai.completion. New in 0.2.1, it also permits outbound
A2A quality explanations under donkey.a2a.quality.explanation, limited to
4096 characters. Gateway explanations may repeat request text and are exported
without sanitization; truncation is not redaction. This A2A behavior is new in 0.2.1.
When off, prompt/completion attributes are excluded by the LLM span allowlist and A2A quality explanation text is omitted by the outbound recorder. Numeric quality scores and the captured evaluation/status values do not require content opt-in. See Outbound A2A spans.
Routing & resilience
The gateway can fail over between providers when one degrades (“Enhanced
Resilience for Intelligent Routing”). It reports what it did with each request
on the response — which provider and model served it, how it routed, and whether
that was a fallback. DDK reads those signals off the shared transport, so
you get them with no framework required — the raw donkey.llm.client() path
benefits just as the deep adapters do.
Every governed call through the SDK’s shared HTTP client exposes them on
donkey.last_call, beside the usage and identity fields:
donkey = Donkey.from_env()
await donkey.openai().responses.create(model="gpt-5.1", input="…")
r = donkey.last_call
r.requested_model # "gpt-5.1" — what you asked for
r.served_model # "gpt-5.1" — or a substitute after failover
r.served_provider # "openai"
r.routing_type # "ModelBased" — or "Semantic" on a semantic-routing proxy
r.fallback # False — True if the gateway failed over
r.substituted # False — served_model != requested_model
r.matched_topic # None — the topic a Semantic proxy matched (else None)
r.routing_score # None — that match's similarity score (else None)On a semantic-routing proxy (routing_type == "Semantic"), the gateway also
reports why it picked a provider: the topic your prompt matched and the
similarity score behind that match. Those land on r.matched_topic /
r.routing_score. On a model-based proxy both are None — the gateway emits no
semantic header, and DDK never fabricates a value for a signal it did not
observe.
They also land on the span (gen_ai.response.model, donkey.routing.type,
donkey.routing.fallback, and — on a semantic route —
donkey.routing.matched_topic / donkey.routing.score) — the most useful thing
to have on hand when latency spikes: it tells an operator whether a slow call was
routed normally or recovered from a degraded provider.
For the complete list of last_call fields — observability status, gateway
identity, routing, and token usage — see the last_call field
reference.
The same record also carries the call’s identity and usage: request_id (the
upstream provider’s id), api_instance_id and environment_id (which gateway
instance served it), and the token counts including cached and reasoning
tokens. The gateway identity example walks
through every field.
When last_call is unavailable
donkey.last_call is populated only when the governed response passes through
the SDK’s shared HTTP client, and only in the context that made the call.
CrewAI routes outside that client: its native OpenAI provider builds its own,
so no response reaches the record and the run’s correlation ID isn’t sent.
LlamaIndex, Microsoft Agent Framework and ADK’s model() send through the
shared client, so a call populates the record in its own context and carries
the run’s correlation ID. A cold read on a Donkey that resolved only these
adapters reports UNOBSERVED, and OBSERVED after a call
(#740 ).
ADK’s gemini() uses the shared client too (see
Native Gemini for reading last_call inside
an ADK run).
For a Donkey that resolved only CrewAI:
r = donkey.last_call
r.status # LastCallStatus.UNAVAILABLE
r.available # False
r.surface # "crewai"This is different from UNOBSERVED, which means the current context has not
yet received a governed response. On an unavailable surface the SDK cannot
observe any response-derived last_call field, including gateway identity,
routing, fallback, and usage. If multiple non-observing adapters were resolved,
surface lists their names.
An agent runner that sends the model call from a task of its own leaves its
caller reading UNOBSERVED, even though the SDK observed the call: the record
lands in that task. The OpenAI Agents SDK’s Runner sends a run’s first turn
that way (and every turn of run_streamed), and ADK’s Runner runs the whole
agent that way. Read last_call in the framework’s model hook, which runs in the
call’s own task: RunHooks.on_llm_end for the
OpenAI Agents SDK, after_model_callback
for ADK. Calling the model directly (Strands’
Agent.invoke_async, AsyncAnthropic.messages.create, a model’s own async
method) records in the caller’s context.
Two behaviours worth knowing
The SDK never double-retries a fallback. DDK retries 502/503/504 with
backoff (a 502/504 on a model call only with
retry_model_calls_on_gateway_errors), but if the gateway already failed over internally, a 503 it marked as
a fallback is not retried again — a second recovery layer stacked on a
working first one just multiplies latency against an outage the gateway already
handled.
Opt in to model determinism. A silent substitution is surfaced passively on
last_call.substituted by default. When a substitution is not acceptable — your
evaluation, cost model and token assumptions are all pinned to one model — opt
into a hard error:
donkey = Donkey.from_env(on_model_substitution="raise")
# raises ModelSubstituted when the served model differs from the requested oneon_model_substitution resolves along the standard precedence (kwarg → env
DONKEY_ON_MODEL_SUBSTITUTION → .donkey-kit.toml → default) and defaults to
"off".
On a model-based routing proxy you address models as provider/model (for
example openai/gpt-5-mini), and the gateway reports the served model without
the prefix (gpt-5-mini), with the provider in its own header. A prefix that
names the served provider is not counted as a difference, so asking for
openai/gpt-5-mini and being served gpt-5-mini by openai is not a
substitution. The same model served by a different provider still is. On a
proxy with more than one provider, a bare name the gateway cannot pin to one
provider is rejected with a 400 before any upstream call, and DDK raises
ModelNotRoutable.
Semantic caching & semantic routing Live
Omni Gateway can answer a request from its semantic cache (a similar
prompt was answered before, so no provider round-trip and no token cost) and
can route semantically (pick the model by matching the prompt to a
topic). Both happen at the gateway. DDK caches nothing and computes no
embeddings itself; it lets you steer the gateway’s cache per request — skip it,
not store a response, or override the TTL or similarity threshold via
donkey.cache(...) — and reports what
happened:
- the cache outcome (
hit,miss,bypass,no-store) and similarity score ondonkey.last_call.cache_status/.cache_scoreand the span (donkey.cache.status/donkey.cache.score); - the matched routing topic and its score, beside the routing fields above.
A cache hit is a verbatim replay with no provider round-trip, so it never
advances the token budget — the replayed usage is not fresh spend.
TypeScript parity Roadmap
The same signals will surface as donkey.lastRouting.servedModel / .fallback
in the TypeScript SDK.
Correlation IDs
Set a per-run id once, and every call inside the block carries it — on the wire, on every span, and on every exception — with nothing threaded through your framework state.
async with donkey.run(id=ticket.id):
await triage_agent.run(ticket)donkey.run(id=…) binds two ids:
- a run id → the
X-Correlation-Idrequest header → the span’sdonkey.correlation_id→DonkeyError.correlation_id. Shared by every call in the block, so a client-side log line joins to the gateway’s own record. - a fresh per-call id → the
X-Donkey-Request-Idrequest header →DonkeyError.call_id. Unique per logical request, stable across that request’s retries, so one call is pinpointable within a run.
The gateway reads the inbound X-Correlation-Id and echoes it verbatim on the
response, so the request and response x-correlation-id are the same value.
X-Donkey-Request-Id is a client-owned per-call id the gateway does not consume.
Both header names are overridable (correlation_header / call_id_header) for a
gateway that expects different ones.
Propagation is contextvar-based, so it reaches through framework nodes (every
LangGraph node, for instance) without threading an argument through every
function, and concurrent runs never leak into each other. It works with or
without OpenTelemetry installed. donkey.run(...) is a dual sync/async
context manager (plain with works too); nested blocks rebind then restore.
Concurrent fan-out. Every call in a fan-out started inside donkey.run()
(asyncio.gather, parallel nodes, tool calls on child tasks) carries exactly
one correlation id. Each task copies the context when it is created, so this
holds even for a task that outlives the block, and a task that opens its own
donkey.run() changes only its own id, never its parent’s or a sibling’s.
Threads do not inherit the context: run them under contextvars.copy_context()
or open a donkey.run() inside the thread. Outside a run nothing is
grouped. Each call gets its own fresh id, so N concurrent calls send N ids.
A refusal leaving the block is re-raised as its typed DonkeyError, whichever
framework wrapped it on the way out, so except PIIDetected works the same
around every framework. See
Typed refusals at the framework boundary;
donkey.run(typed_refusals=False) turns it off.
Across agent hops
An agent that serves requests can join its caller’s run instead of starting
a fresh one. donkey.run_from_headers() reads the configured correlation header
from the inbound request, adopts the caller’s id when it is well-formed
([A-Za-z0-9._:-], at most 128 characters), and mints a fresh one otherwise. A
malformed id is never forwarded. Every governed call in the block then carries
it, exactly as inside donkey.run(id=…).
@app.post("/tasks")
async def handle(request: Request, task: Task):
async with donkey.run_from_headers(request.headers):
return await agent.run(task)For an outbound hop the governed transport does not carry, such as another
agent’s endpoint or an MCP server reached on your own HTTP client, pass
donkey.correlation_headers(). It returns the bound run id under the configured
header, or {} outside a run, and never the gateway credentials.
async with donkey.run(id=ticket.id):
await agents.post(sub_agent_url, json=task, headers=donkey.correlation_headers())The sub-agent’s spans and the gateway’s record then share one id end to end. The
GenAI span inside run_from_headers() records donkey.correlation_id.source:
adopted when the caller supplied the id, minted when it did not. A served
A2A task applies the same rule on its own.
Run ledger
donkey.last_call keeps only the most recent call. A donkey.run() block also
keeps a run ledger: every governed model call made in the run, every refusal
the gateway returned, and the run’s token totals. Read it inside the block at
donkey.current_run. Keep the scope to read it after the block ends:
from donkey_kit import PIIDetected
scope = donkey.run(id=ticket.id)
async with scope as run_id: # still the run's id, a str
await triage_agent.run(ticket)
run = donkey.current_run
if any(r.is_a(PIIDetected) for r in run.refusals):
... # redact the later prompts
ledger = scope.ledger # still readable after the block
ledger.call_count # every governed model call in the run
ledger.usage.total_tokens # summed across them, cached and reasoning tokens too
[c.served_model for c in ledger.calls] # each entry is that call's last_call record| Field | What it holds |
|---|---|
calls | Each call’s last_call record, oldest first: served model and provider, fallback, request id, usage, semantic-cache and semantic-routing outcome |
refusals | A RunRefusal for each policy refusal (PIIDetected, TokenBudgetExceeded, …) the gateway returned on a call in the run, even when a framework caught it, and for any ModelSubstituted the SDK raised: its kind, policy, status_code, the three ids and, for PII, the entity types. Test it with r.is_a(PIIDetected) |
usage | A RunUsage with the run’s input_tokens, output_tokens, total_tokens, cached_tokens, cache_write_tokens and reasoning_tokens, and calls_with_usage |
call_count, refusal_count | Totals for the whole run |
dropped | Calls and refusals the cap dropped from calls and refusals |
available, surface | False, with the adapter names, when nothing used can feed the ledger |
Each read returns a frozen snapshot, so read again for newer state. The ledger
holds metadata only, never prompt or completion text. A refusal is recorded as a
RunRefusal, not the exception, because the exception’s .response keeps the
gateway’s body, which can echo the values a policy blocked.
- Scope. The ledger follows the run’s correlation id. Framework-spawned
tasks append to their run’s ledger, concurrent runs never see each other’s,
and a nested
donkey.run()keeps its own and restores the outer one on exit. Appends are thread-safe for threads run undercontextvars.copy_context(). - Bounded memory.
callsandrefusalseach keep the newest 1000 entries (donkey.run(ledger_max_entries=…)). Older ones are counted indropped. The totals keep counting past the cap, so a long run’s cost stays exact. - Same reach as
last_call. Only responses through the SDK’s shared HTTP client reach it. When every adapter used routes outside that client (CrewAI today),availableisFalseandsurfacenames them: the ledger is unavailable, not empty. See Whenlast_callis unavailable.
The decorator form
When a whole function should be one run, @donkey.governed is the decorator
equivalent of wrapping its body in donkey.run():
@donkey.governed(team="support")
async def handle_ticket(ticket):
await triage_agent.run(ticket)Each call opens its own run — a fresh run/correlation id — and binds the
optional per-run cost tags, the OTel span, and typed refusals, exactly the scope
donkey.run() establishes. It wraps both sync and async callables and is
usable bare (@donkey.governed) or parametrised. There is deliberately
no id=: pinning one id across every call would collapse unrelated runs into
a single correlation, so when you need a specific id, use donkey.run(id=…)
directly.
Cost-attribution tags
A small, fixed set of tags (team, project, env, enduser.id) set once
and recorded on every model call’s span:
donkey = Donkey.from_env(team="support", project="triage-v2", env="prod")
async with donkey.run(id=ticket.id, enduser_id=agent_user.id):
await triage_agent.run(ticket)The tags resolve in this order: Donkey.from_env(team=…) kwargs, then
DONKEY_COST_* env vars, then a [donkey.cost] table in the config files (see
Configuration). Per-run
overrides layer on top: donkey.run(team=…, project=…, env=…, enduser_id=…)
wins per field for its block, and the rest fall back to the configured
tags. The key set is fixed: an unknown dimension is a configuration error,
never a silently dropped tag. Values are validated (fixed keys, bounded
length), so nobody stuffs a JSON blob into a span attribute or header.
Where the tags go
| Carrier | Names | When | Goes to |
|---|---|---|---|
| Span attributes | donkey.cost.team, donkey.cost.project, donkey.cost.env, donkey.cost.enduser.id | Always, for each tag that is set, on every donkey.llm.chat span. Needs telemetry on (the default) and OpenTelemetry installed. The SDK’s shared client opens the span, so CrewAI, which has only header injection, records none. | Your OpenTelemetry pipeline: the OTLP exporter when OTEL_EXPORTER_OTLP_ENDPOINT is set, or any TracerProvider your process installed |
| Request headers | X-Anypoint-Cost-Team, X-Anypoint-Cost-Project, X-Anypoint-Cost-Env, X-Anypoint-Cost-Enduser-Id (renamable with cost_*_header) | Only with send_cost_headers / DONKEY_SEND_COST_HEADERS=true. Off by default. | Model requests to the LLM proxy only. See the note below. |
Each header carries the tag’s value unchanged (for example
X-Anypoint-Cost-Enduser-Id: user-42). A tag that isn’t set is sent under
neither carrier.
The Anypoint LLM Gateway doesn’t read cost tags from request headers: it meters cost from token usage per API instance and consuming client application. So the span attributes are the carrier that matters. Turn the headers on only when something of your own reads them.
With send_cost_headers on, the headers go only on model requests to the
LLM proxy. The connected-app token request and other Anypoint
control-plane requests never carry them, so the end-user ID doesn’t reach
those hosts. Adapters with transport injection also send per-run
donkey.run(...) values; adapters that get only a default_headers snapshot
send the configured tags.
The end-user ID reaches your trace backend regardless of
send_cost_headers. If enduser_id is set, every model-call span carries
donkey.cost.enduser.id, and any exporter you’ve configured receives it. To
keep it out of traces, don’t set enduser_id (DONKEY_COST_ENDUSER_ID,
"enduser.id" in [donkey.cost], or donkey.run(enduser_id=…)), drop the
attribute in your collector, or turn spans off with DONKEY_TELEMETRY=false.
Per-request attribution
One process often serves several tenants or teams: a shared agent platform, a
multi-tenant SaaS agent, or one job that spans two cost centres. Keep one
Donkey (one client, pool, auth cache and budget) and say per request which
group a call belongs to:
async def handle(tenant: str, prompt: str):
with donkey.attribution(business_group=tenant):
return await model.ainvoke(prompt)donkey.attribution(business_group=…, team=…, project=…, env=…, enduser_id=…)
attributes every governed call in its block, on both the async and the blocking
client. The business group replaces the configured business_group for the
block. The precedence is this scope, then the Donkey config, then
DONKEY_BUSINESS_GROUP, then the config file. The cost tags merge per field
over any already bound by an enclosing block.
It doesn’t start a run, so the correlation ID stays the same and one run can span two cost centres:
async with donkey.run(id=job.id):
with donkey.attribution(business_group="risk-eu"):
await score(job)
with donkey.attribution(business_group="risk-us"):
await score(job)donkey.run(business_group=…) and @donkey.governed(business_group=…) take
the same argument when the group belongs to the whole run. Like the run ID, the
binding is contextvar-based: it reaches framework-spawned tasks, concurrent
tasks never see each other’s group, and nested blocks restore the outer value
on exit. The value is validated like a cost tag, and an invalid one raises
ConfigError where you set it.
Adapters that get only a default_headers snapshot don’t follow the
scope. They captured the configured business group when they were built, so
they keep sending it. Adapters with transport injection send the scoped value.
The header name, X-Anypoint-Business-Group, is still an unverified
placeholder; see Configuration.
The question this answers
Finance asks what the support agent cost last month versus the HR bot. Without
tags, both agents share one client_id and there is no way to split the bill.
With tags it is a group-by.
For compliance — “prove the HR bot’s answer to user X on date Y went through
the content-safety policy” — the correlation ID on the log line joins to the
gateway record, and the span carries donkey.cost.enduser.id and donkey.policy.type. An
EU AI Act Article 12 log request becomes one query, not an investigation.
Debug logging
Spans tell you what a call did. When you need to see what the SDK did for
you along the way, such as a retry, a token refresh or a fallback, turn on its
DEBUG logging. It uses the standard library logging module, with one logger
per module under the donkey_kit package logger:
import logging
logging.basicConfig() # or your own handlers
logging.getLogger("donkey_kit").setLevel(logging.DEBUG)A retried 503 then reads:
DEBUG:donkey_kit.core.transport.observe:GET https://proxy.example/v1/chat/completions returned 503; retry 1 of 3 in 0.48sThe SDK attaches only a NullHandler to donkey_kit and never configures a
handler, level or format of its own, so nothing prints until your application
does. That also means you can narrow it. Set the level on one module logger,
such as donkey_kit.core.auth, instead of the package.
| What is logged at DEBUG | Logger |
|---|---|
| Each retry: the status, which retry it is, and the delay | donkey_kit.core.transport.observe |
| A retryable status that is not retried: after a gateway routing fallback, a 502/504 on a model call (see Retry behaviour), or once retries run out | donkey_kit.core.transport.observe |
| A 401 on the async client, followed by one token refresh and re-send | donkey_kit.core.transport.async_client |
| A connected-app token fetch (the token URL only) and token invalidation | donkey_kit.core.auth |
An auth provider in a ChainedAuth failing over to the next one (type names only) | donkey_kit.core.auth |
The OTLP export bootstrap: inert, kept your TracerProvider, or installed (global or DDK-scoped) | donkey_kit.core.telemetry |
How many tool descriptors a ToolSet exposes after its filter, out of the total | donkey_kit.tools.session |
| A rate-limit reset header that was ignored as negative or out of range, naming the budget window (never the value) | donkey_kit.core._window |
| A refusal translator that failed and was skipped, or a refusal capture that could not tag a framework error (type names only, no hook representation or traceback) | donkey_kit.core.refusals |
| Other best-effort steps that failed and were skipped (with the traceback), e.g. recording usage on a span | the module that skipped it |
A log record never carries a credential. Records name a request by method, scheme, host and path only. They never include a header value, token, client secret, query string or request body, and an auth failure is logged by exception type, not message. A unit test pins this against every credential the SDK sends: the bearer token, client id and client secret.
If every ChainedAuth provider fails, the final AuthError names only the
last exception type, not its message, and suppresses exception chaining.
Refusal-handler failures likewise omit callback messages and tracebacks; the
donkey_kit.core.reactions logger records only the handler label and exception
type at WARNING, and the original refusal still propagates.
The SDK does not use WARNING log records for configuration problems. A one-time
problem you have to fix is a Python warning instead, raised through
warnings.warn with its own category, so you can filter it or make it an
error with the warnings module. For example, an OTLP endpoint set without its
exporter installed raises donkey_kit.core.telemetry.TelemetryExportWarning
(see Zero-config export).