Skip to Content
These docs describe donkey-kit v0.2.1. Release notes
ExamplesGeneralBudget & pacing

Live

Budget & pacing

The gateway reports your token budget only in-band, on response headers — there is no endpoint to ask how much is left. donkey.budget reads those headers for you, so the window is an object rather than a header you parse. The useful half is pacing: pace(reserve=…) refuses locally before issuing a request that would cross your reserve, turning a 429 you would have to recover from into an exception you chose to raise, and wait_for_reset() sleeps once until the window rolls over.

ExampleShowsNeeds
Narrative demo 03A cold process knowing nothing, the window updating per response, pace(reserve=0.05) raising BudgetReserveReached, wait_for_reset(), and the terminal 429Nothing (simulator)
OpenAI script 05pace(reserve=) letting the first call through, stopping the second, then wait_for_reset()Proxy credentials

Run it

make demo N=03
Expected output
════════════════════════════════════════════════════════════════════════════════════════ Demo 03 — budget and pacing The token window as an object, and refusing to cross it before the gateway does. ════════════════════════════════════════════════════════════════════════════════════════ Run context ─────────── target mock proxy base_url http://127.0.0.1:8080/ output masking on credentials fake — the simulator enforces no auth [1] A new process knows nothing until its first call returns before any call remaining=None limit=None used=unobserved Every field is None rather than zero. An unobserved budget reports 'I don't know', because reporting 0 remaining would be a lie that stops an agent that could have run. [2] Each response updates the window, with no code from you await client.responses.create(model=..., input=...) donkey.budget.remaining # already up to date after call 1 remaining=97500 limit=100000 used=2.5% after call 2 remaining=97000 limit=100000 used=3.0% after call 3 remaining=96500 limit=100000 used=3.5% observed_at 2026-09-24 09:27:23.806252+00:00 reset_at 2026-09-24 09:28:23.806252+00:00 A live 200 carries this window as the prose header x-llm-proxy-ratelimit — that sentence is live-verified. The numeric x-token-* trio is verified on the 429. The simulator synthesises a decreasing window in the same prose shape, so the numbers above are illustrative; the parse path is not. [3] pace(reserve=…) refuses before the request goes out Rather than issue the 200 calls it would take to drain the simulator's window, we let the budget observe a response that says we are already at 96% — the same code path a real near-exhausted window takes. async with donkey.budget.pace(reserve=0.05): await enrich(batch) # never runs if the reserve is crossed observed remaining=4000 limit=100000 used=96.0% reset_at 2026-09-24 09:27:24.806704+00:00 PASS BudgetReserveReached — the request was never issued fraction_used 96.0% reserve 5.0% reset_at 2026-09-24 09:27:24.806704+00:00 BudgetReserveReached is deliberately NOT a PolicyViolation. A refusal is the gateway saying no and is terminal; this is your own client-side signal, raised locally, that you are expected to recover from. try: async with donkey.budget.pace(reserve=0.05): await enrich(batch) except BudgetReserveReached: await donkey.budget.wait_for_reset() # one sleep, never a spin loop after wait_for_reset remaining=4000 limit=100000 used=96.0% PASS wait_for_reset() slept until reset_at — one sleep, never a spin loop The local object is still the last observation. Waiting does not invent a fresh window; the next call is what refreshes remaining / limit / reset_at. That is the same in-band rule as act 1. [4] And if you do cross it, the 429 is terminal classified as TokenBudgetExceeded retry_after 41.728 PASS The transport never retried it — retrying only burns the same window. This is the scenario the conformance suite checks other people's agents for, because retrying a budget refusal is the single most common way an agent turns one refusal into a rate-limit spiral. See demo 05. ────────────────────────────────────────────────────────────────────────────────────────
python "demos/human-made/openai/05 - budget_and_pacing.py" # needs proxy credentials

On a live gateway the window arrives only when the token rate limit policy is applied to the proxy. The simulator synthesises a decreasing window so pacing can run locally; its happy-path numbers are illustrative, the parse path is not.

Key code

Pacing, and recovering from it (narrative demo 03, act 3):

try: async with donkey.budget.pace(reserve=0.05): await enrich(batch) except BudgetReserveReached: await donkey.budget.wait_for_reset() # one sleep, never a spin loop

Against a live proxy, the first call is unobserved so pace() lets it through and the window arrives in-band; the second trips the reserve (OpenAI script 05):

async with donkey.budget.pace(reserve=0.99999): response = await client.responses.create( model="gpt-4o", input="Say hello in exactly three words.", ) budget = donkey.budget print("after request 1, budget remaining is", budget.remaining) print("after request 1, budget fraction_used is", budget.fraction_used) try: async with donkey.budget.pace(reserve=0.99999): response = await client.responses.create( model="gpt-4o", input="Say hello in exactly three words.", ) except BudgetReserveReached as exc: print("stopped locally [in-script]", exc.fraction_used, exc.reserve)

An unobserved budget reports None for every field, never 0 — reporting 0 remaining would stop an agent that could have run. BudgetReserveReached is deliberately not a PolicyViolation: a refusal is the gateway saying no and is terminal, while this is a local signal you are expected to recover from. If you do cross the window, the resulting TokenBudgetExceeded is not retried by the transport — retrying only burns the same window.

The request window

When the gateway’s rate-limiting policy has exposeHeaders: true, donkey.budget.requests (a RequestWindow) tracks the request count the same way, and pace() refuses when either window has reached the reserve. BudgetReserveReached.window names which one ("tokens" or "requests"), so pass it on to wait_for_reset():

while True: try: async with donkey.budget.pace(reserve=0.05): await enrich(batch) except BudgetReserveReached as exc: if exc.reset_at is None: raise await donkey.budget.wait_for_reset(window=exc.window) continue break

On a small request limit a fractional reserve is coarse (with 3 requests, reserve=0.05 behaves like 0.0), and pace() is not a concurrency limiter. Run it locally with donkey mock --scenario request_limit:limit=3,window=60s. The refusal itself is RequestRateLimitExceeded, terminal like TokenBudgetExceeded.

Concurrent calls

A parallel fan-out (asyncio.gather, parallel graph nodes, threads on the sync client) shares one donkey.budget. Responses can be handled in any order, but the budget never moves backwards: each window keeps the freshest observation, ordered by when that response’s headers arrived, not by when its hooks finished. pace() decides on one consistent snapshot, so tasks entering it together get the same answer.

pace() is an admission check, not a reservation. It cannot see what calls already in flight will spend, so a fan-out that enters together is admitted together. Size reserve to cover the width of your fan-out.

Learn more: Budget & pacing

Source: narrative demo 03  · script 05 

Last updated on