Provider TPM Pools

Technical reference. This page is for integrations, advanced configuration, or platform operations. For portal tasks, start with the user guide.

Provider TPM pools are the per-(organization, provider, model) throttle that the runtime-gateway acquires against on every LLM call. They protect upstream providers from sustained over-rate traffic AND short bursty spikes, both at the same time. The pool was added in ADR 0063 alongside the org daily-cost pool; the two are typed siblings — TPM is a sliding-window counter (no commit phase), cost is a two-phase reservation.

What this surface does

Every LLM call passes through multi_pool_acquire in the runtime-gateway, which acquires a TPM reservation BEFORE acquiring the cost reservation. The acquire reserves prompt_tokens against two simultaneous windows for the same (org, provider, model) triple:

• a sustained 60-second window capped by rate_tpm
• a 5-second burst window capped by burst_tpm

If either window would exceed its cap, the acquire returns 429 with tpm_sustained_throttled or tpm_burst_throttled. The runtime-gateway propagates the 429 back to the caller as a structured decision payload; no provider call is made.

There is no commit phase. Once the call completes (success, error, or egress-blocked), the runtime-gateway calls the release route which removes the reservation entry from the sliding-window ZSETs. If the runtime-gateway crashes before release, the reservation entry’s TTL ages it out — the windows self-heal within max(rate_window, burst_window) + 5s at worst.

The pool fails open on Redis outage (spec §11.4 row 2): a Redis-down acquire returns 200 with decision.governor_unavailable=true so traffic is not held hostage to Redis health. The cost pool fails closed instead because cost requires Redis atomicity for the reservation snapshot.

Admin routes

All five routes live under /v1/governance/organizations/{org_id}/providers/{provider}/models/{model}/... and are gated by Permission.GOVERNANCE_POOL_VIEW (reads) or Permission.GOVERNANCE_POOL_MANAGE (writes), both at organization scope.

Two additional internal routes (POST /v1/governance/tpm/acquire, DELETE /v1/governance/tpm/reservations/{request_id}) are consumed only by the runtime-gateway and require an internal service / system / workload actor — user JWTs are rejected with 403 before the route body runs.

Request / response shapes

PUT tpm-config

{
  "rate_tpm": 60000,
  "burst_tpm": 90000,
  "rate_window_seconds": 60,
  "burst_window_seconds": 5
}

Returns:

{
  "config_id": "8c3a1b9e-0cae-4a63-8ad8-50ee3d5e60f1",
  "pool_id": "provider_tpm:{org_id}:{provider}:{model}",
  "rate_tpm": 60000,
  "burst_tpm": 90000
}

POST tpm-override

{
  "tpm_ceiling": 100000,
  "tpm_rate_ceiling": 80000,
  "reason": "release-2026-04-27: bulk re-evaluation",
  "expires_at": "2026-04-27T22:00:00Z"
}

At least one of tpm_ceiling / tpm_rate_ceiling must be set; both null returns 400. expires_at more than 24h in the future returns 400 override_ttl_too_long.

POST tpm-reservations/reset

{ "reason": "incident-2026-04-27: stranded keys after redis-failover" }

Returns the count of removed reservation keys plus the cleared ZSETs.

Example invocations

# Read the active config
curl -H "Authorization: Bearer $TOKEN" \
  "https://control-plane/v1/governance/organizations/$ORG_ID/providers/{provider}/models/{model}/tpm-config"

# Set durable limits
curl -X PUT \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"rate_tpm": 60000, "burst_tpm": 90000}' \
  "https://control-plane/v1/governance/organizations/$ORG_ID/providers/{provider}/models/{model}/tpm-config"

# Apply a 2-hour override
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"tpm_ceiling\": 100000, \"reason\": \"release bulk run\", \"expires_at\": \"$(date -u -d '+2 hours' +%Y-%m-%dT%H:%M:%SZ)\"}" \
  "https://control-plane/v1/governance/organizations/$ORG_ID/providers/{provider}/models/{model}/tpm-override"

# Reset the sliding windows (incident-only)
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"reason": "incident-2026-04-27: redis-failover stranded entries"}' \
  "https://control-plane/v1/governance/organizations/$ORG_ID/providers/{provider}/models/{model}/tpm-reservations/reset"

The same operations are wrapped by the CLI: orqestractl pool tpm get, orqestractl pool tpm set, orqestractl pool tpm override, orqestractl pool tpm reset.

Defaults and configuration

Fallback values used only when no active limits row exists for the (org, provider, model) triple:

A workspace pinned at the env defaults will see fallback behaviour until an operator persists a limits row via PUT.

Operator alerts

The TPM pool exposes metrics for operator observability and fast triage:

• tpm_acquires_total{provider, model, outcome} — counter, every acquire attempt with the resolved outcome label (allowed, sustained_throttled, burst_throttled, policy_denied, override_allowed).
• tpm_throttle_events_total{provider, model, window} — counter, throttle-only rollup with the window label (sustained / burst) for fast triage.
• tpm_acquire_duration_seconds{provider, model, outcome} — histogram (runtime-gateway side) for latency budgeting.

Operators who want a dedicated alert can add one against rate(tpm_throttle_events_total[5m]) per (provider, model). The default deployment ships without one because TPM throttling is a designed back-pressure signal — sustained throttling is a workload-tuning issue, not an incident.

Related

• Org Daily Cost — the cost-side companion to TPM that frozen-snapshots pricing into a two-phase commit.
• Pricing Table — the platform-scoped (provider, model) pricing rows that TPM pools refer to indirectly via the cost pool.
• Cost & Pricing Recovery Runbook  — operator runbook for stuck reservations, pricing audits, anomaly investigation, and snapshot reruns.
• ADR 0063 — Provider TPM and organization daily cost pools .