Live
Budget & pacing
The gateway reports your token budget only in-band, on response headers —
there is no endpoint to ask how much is left. donkey.budget reads those
headers for you, so the window is an object rather than a header you parse.
The useful half is pacing: pace(reserve=…) refuses locally before issuing a
request that would cross your reserve, turning a 429 you would have to recover
from into an exception you chose to raise, and wait_for_reset() sleeps once
until the window rolls over.
| Example | Shows | Needs |
|---|---|---|
| Narrative demo 03 | A cold process knowing nothing, the window updating per response, pace(reserve=0.05) raising BudgetReserveReached, wait_for_reset(), and the terminal 429 | Nothing (simulator) |
| OpenAI script 05 | pace(reserve=) letting the first call through, stopping the second, then wait_for_reset() | Proxy credentials |
Run it
make demo N=03════════════════════════════════════════════════════════════════════════════════════════
Demo 03 — budget and pacing
The token window as an object, and refusing to cross it before the gateway does.
════════════════════════════════════════════════════════════════════════════════════════
Run context
───────────
target mock
proxy base_url http://127.0.0.1:8080/
output masking on
credentials fake — the simulator enforces no auth
[1] A new process knows nothing until its first call returns
before any call remaining=None limit=None used=unobserved
Every field is None rather than zero. An unobserved budget reports 'I don't know',
because reporting 0 remaining would be a lie that stops an agent that could have run.
[2] Each response updates the window, with no code from you
await client.responses.create(model=..., input=...)
donkey.budget.remaining # already up to date
after call 1 remaining=97500 limit=100000 used=2.5%
after call 2 remaining=97000 limit=100000 used=3.0%
after call 3 remaining=96500 limit=100000 used=3.5%
observed_at 2026-09-24 09:27:23.806252+00:00
reset_at 2026-09-24 09:28:23.806252+00:00
A live 200 carries this window as the prose header x-llm-proxy-ratelimit — that
sentence is live-verified. The numeric x-token-* trio is verified on the 429. The
simulator synthesises a decreasing window in the same prose shape, so the numbers
above are illustrative; the parse path is not.
[3] pace(reserve=…) refuses before the request goes out
Rather than issue the 200 calls it would take to drain the simulator's window, we let
the budget observe a response that says we are already at 96% — the same code path a
real near-exhausted window takes.
async with donkey.budget.pace(reserve=0.05):
await enrich(batch) # never runs if the reserve is crossed
observed remaining=4000 limit=100000 used=96.0%
reset_at 2026-09-24 09:27:24.806704+00:00
PASS BudgetReserveReached — the request was never issued
fraction_used 96.0%
reserve 5.0%
reset_at 2026-09-24 09:27:24.806704+00:00
BudgetReserveReached is deliberately NOT a PolicyViolation. A refusal is the gateway
saying no and is terminal; this is your own client-side signal, raised locally, that
you are expected to recover from.
try:
async with donkey.budget.pace(reserve=0.05):
await enrich(batch)
except BudgetReserveReached:
await donkey.budget.wait_for_reset() # one sleep, never a spin loop
after wait_for_reset remaining=4000 limit=100000 used=96.0%
PASS wait_for_reset() slept until reset_at — one sleep, never a spin loop
The local object is still the last observation. Waiting does not invent a fresh
window; the next call is what refreshes remaining / limit / reset_at. That is the same
in-band rule as act 1.
[4] And if you do cross it, the 429 is terminal
classified as TokenBudgetExceeded
retry_after 41.728
PASS The transport never retried it — retrying only burns the same window.
This is the scenario the conformance suite checks other people's agents for, because
retrying a budget refusal is the single most common way an agent turns one refusal
into a rate-limit spiral. See demo 05.
────────────────────────────────────────────────────────────────────────────────────────python "demos/human-made/openai/05 - budget_and_pacing.py" # needs proxy credentialsOn a live gateway the window arrives only when the token rate limit policy is applied to the proxy. The simulator synthesises a decreasing window so pacing can run locally; its happy-path numbers are illustrative, the parse path is not.
Key code
Pacing, and recovering from it (narrative demo 03, act 3):
try:
async with donkey.budget.pace(reserve=0.05):
await enrich(batch)
except BudgetReserveReached:
await donkey.budget.wait_for_reset() # one sleep, never a spin loopAgainst a live proxy, the first call is unobserved so pace() lets it through
and the window arrives in-band; the second trips the reserve (OpenAI script 05):
async with donkey.budget.pace(reserve=0.99999):
response = await client.responses.create(
model="gpt-4o",
input="Say hello in exactly three words.",
)
budget = donkey.budget
print("after request 1, budget remaining is", budget.remaining)
print("after request 1, budget fraction_used is", budget.fraction_used)
try:
async with donkey.budget.pace(reserve=0.99999):
response = await client.responses.create(
model="gpt-4o",
input="Say hello in exactly three words.",
)
except BudgetReserveReached as exc:
print("stopped locally [in-script]", exc.fraction_used, exc.reserve)An unobserved budget reports None for every field, never 0 — reporting 0
remaining would stop an agent that could have run. BudgetReserveReached is
deliberately not a PolicyViolation: a refusal is the gateway saying no and is
terminal, while this is a local signal you are expected to recover from. If you
do cross the window, the resulting TokenBudgetExceeded is not retried by the
transport — retrying only burns the same window.
The request window
When the gateway’s rate-limiting policy has exposeHeaders: true,
donkey.budget.requests (a RequestWindow) tracks the request count the same
way, and pace() refuses when either window has reached the reserve.
BudgetReserveReached.window names which one ("tokens" or "requests"), so
pass it on to wait_for_reset():
while True:
try:
async with donkey.budget.pace(reserve=0.05):
await enrich(batch)
except BudgetReserveReached as exc:
if exc.reset_at is None:
raise
await donkey.budget.wait_for_reset(window=exc.window)
continue
breakOn a small request limit a fractional reserve is coarse (with 3 requests,
reserve=0.05 behaves like 0.0), and pace() is not a concurrency limiter.
Run it locally with donkey mock --scenario request_limit:limit=3,window=60s.
The refusal itself is RequestRateLimitExceeded, terminal like
TokenBudgetExceeded.
Concurrent calls
A parallel fan-out (asyncio.gather, parallel graph nodes, threads on the sync
client) shares one donkey.budget. Responses can be handled in any order, but
the budget never moves backwards: each window keeps the freshest
observation, ordered by when that response’s headers arrived, not by when its
hooks finished. pace() decides on one consistent snapshot, so tasks entering
it together get the same answer.
pace() is an admission check, not a reservation. It cannot see what calls
already in flight will spend, so a fan-out that enters together is admitted
together. Size reserve to cover the width of your fan-out.
Learn more: Budget & pacing
Source: narrative demo 03 · script 05