Backroom · Ditto Ops

SN118 inference

Inference & benchmark runtime

Control hosted inference budgets, chat and embedding admission, v10 case concurrency, and relay delay fingerprints. Changes are lease-stamped from an append-only audited revision. Case concurrency overlaps /run against the process-wide inference URL.

Platform managed

Current inference and runtime policy

What one scoring ticket may spend, how parallel and per-minute inference traffic may be, and how a newly issued v10 lease schedules its cases.

Revision: 0
Source: Shipped default
Propagation: Admission ≤5s · runtime next lease

The two budgets are stamped onto a grant when it is minted, so a change here governs the next lease and can never retroactively exhaust a run already in flight. Chat and embedding concurrency and request-per-minute limits are enforced at admission instead — which is what makes lowering a per-ticket limit safe to pull mid-run: a decline is answered with 503 and Retry-After, so a validator backs off and continues rather than discarding the run.

Chat token budget (1–200,000,000)

Chat tokens (prompt + completion) one scoring ticket may spend. This is the allowance that binds in practice — raising the request budget alone left the heaviest strategies failing in exactly the same place. A cap, not a spend: raising it changes only which runs are permitted to finish.

Chat request budget (1–32,768)

Chat completions one ticket may spend in total. The bound that survives a pathological loop of tiny requests, which the token budget would absorb slowly.

Chat · per ticket (1–512)

Concurrent hosted chat requests one scoring ticket may hold.

Chat · per validator (1–512)

Concurrent hosted chat requests summed over one validator’s grants.

Chat · fleet (1–512)

Concurrent hosted chat requests across the fleet. Live within five seconds without a relay restart.

Embedding · per ticket (1–512)

Concurrent hosted embedding requests one ticket may hold. The emergency brake: lowering it takes effect fleet-wide on the next admission.

Embedding · per validator (1–512)

Summed over one validator’s grants.

Embedding · fleet (1–512)

Across the whole fleet. Enforced by a cross-grant aggregate, so it is best-effort under a simultaneous burst — size it as a load-shedding backstop, not an exact valve.

Chat RPM · per ticket (1–100,000)

Hosted chat starts one ticket may begin in any rolling minute. This is the rail that bound 8-wide v11 runs at 240/min while concurrency peaks looked idle. Same 503 as a full lane.

Chat RPM · per validator (1–100,000)

Hosted chat starts summed over one validator’s grants per rolling minute.

Chat RPM · fleet (1–100,000)

Hosted chat starts across the fleet per rolling minute.

Embedding RPM · per ticket (1–100,000)

Hosted embedding starts one ticket may begin in any rolling minute.

Embedding RPM · per validator (1–100,000)

Hosted embedding starts summed over one validator’s grants per rolling minute.

Embedding RPM · fleet (1–100,000)

Hosted embedding starts across the fleet per rolling minute.

V10 benchmark runtime

Stamped onto newly issued v10+ leases. In-flight leases keep their stamp. v9 has no stamp and uses the scorer default (4). A stored revision of 1 stays live until you write a new whole-object policy. Values above 16 need every validator on a release that accepts 1–64; older validators reject such leases outright.

V10 case concurrency (1–64)How many /run calls one ticket may overlap. Default 4. Miners keep one process-wide inference URL.Relay delay fingerprintShadow injects a secret deterministic hold and records evidence only inside confirmation case windows; ordinary scored runs no longer open case windows, so it is a no-op there. It never changes scores.Minimum delay (ms)Maximum delay (ms)

Operator reasonType to confirmAPPLY INFERENCE CONCURRENCY SETTINGS