Evaluation Gates

Technical reference. This page is for integrations, advanced configuration, or platform operations. For portal tasks, start with the user guide.

Evaluation gates are quality checkpoints that block release state transitions until required evaluation suites pass. They provide automated, evidence-based confidence before a release reaches human reviewers or goes live — ensuring that every behavioral change in a workload is validated against defined expectations.

What Are Evaluation Gates

An evaluation gate is a policy-enforced checkpoint in the release lifecycle. Before a release can advance to the next state, the gate checks whether all required evaluation suites have run and met their pass-rate thresholds. If any required suite is missing or below threshold, the transition is rejected with a structured error.

Default gate position: The gate fires before the review_ready → approved transition. Human reviewers only see releases that have already cleared automated evaluation.

Configurable gate position: The gate can be moved to the approved → active transition instead, so evaluation runs closer to the moment of activation.

Default (gate before approval):

  draft --> review_ready --(eval gate)--> approved --> active

Alternative (gate before activation):

  draft --> review_ready --> approved --(eval gate)--> active

When no evaluation configuration exists for a workload (no policy bundle and no workload-level eval_config), the gate is open and transitions proceed without evaluation checks.

Evaluation Suites

An evaluation suite is a governed asset with asset_kind: "evaluation_suite". It defines the test cases and scoring rules used to measure a release.

Visibility and ownership

Evaluation selectors show suites owned by the active workspace and platform-owned global built-ins. A global built-in is shared only when the platform record has no workspace, no namespace, and the platform global-ownership marker; a tenant cannot make a suite global by omitting a workspace. Release evaluation, candidate testing, and Compare apply the same rule and pin the selected suite version before execution.

Global built-ins are read-only to tenant users. They may be selected for an evaluation or comparison, but they do not appear in mutation pickers such as Add to suite, which creates a new version. Add or edit cases only on a suite owned by your workspace.

The control plane revalidates the pinned suite and version before execution, gate decisions, evidence, and metrics. A stale or malformed reference is excluded rather than treated as global; when that reference was required, the gate remains blocked as if the required result were missing.

Suite Structure

Test Cases

Each test case specifies an input, an expected output object, and how to compare the actual output against the expected. A case can also link back to the governed asset it evaluates.

Evaluation suites are bounded at authoring and again before execution: a suite contains at most 256 cases, a regex case contains at most 32 expected fields, and a suite contains at most 256 regex fields in total. Regex patterns are limited to 512 characters and each expected value is a string of at most 512 characters. The runtime uses a timeout-capable engine with a 0.25-second per-match deadline and one 256-match/1-second cumulative budget for a complete candidate or comparison dispatch. Invalid, unsafe, oversized, timed-out, or over-budget regex cases fail closed; the platform never falls back to an unbounded regular-expression engine. Compare requests are also capped at 256 cohorts and reject a suite-pattern/cohort cross-product above that budget before creating or dispatching a run.

{
  "asset_kind": "evaluation_suite",
  "slug": "refund-policy-accuracy",
  "test_cases": [
    {
      "id": "tc-001",
      "input": { "user_message": "Can I return a product after 60 days?" },
      "expected_output": { "text": "30-day return policy" },
      "match_mode": "contains",
      "subject_kind": "prompt_template",
      "subject_ref": "refund-policy-prompt",
      "subject_version_ref": "refund-policy-prompt@v3"
    },
    {
      "id": "tc-002",
      "input": { "user_message": "What is the refund window?" },
      "expected_output": { "text": ".*30.*(day|days).*" },
      "match_mode": "regex"
    }
  ],
  "default_threshold": 0.9
}

The suite pass rate is the mean score across executable test cases. exact, contains, and regex produce deterministic binary scores. Do not author semantic cases yet; they validate for compatibility but fail closed at runtime.

Federated Configuration

Evaluation gate behavior is configured at two levels: the policy bundle (operator floor) and the workload (author overrides). The two levels are merged at gate-check time.

Policy Bundle Settings

A policy bundle can include an eval_policy block:

{
  "eval_policy": {
    "gate_position": "before_approval",
    "min_pass_rate": 0.85,
    "required_suite_kinds": ["accuracy", "safety"]
  }
}

Workload eval_config

A workload can override policy settings in its eval_config. Overrides can only be stricter — they cannot relax what the policy bundle requires:

{
  "eval_config": {
    "suites": [
      {
        "suite_slug": "refund-policy-accuracy",
        "threshold": 0.95
      },
      {
        "suite_slug": "tone-safety",
        "threshold": 0.88
      }
    ]
  }
}

Merge Rules

When both a policy bundle and a workload eval_config are present, the runtime applies the following merge rules:

This means a workload can require additional suites or raise thresholds, but cannot skip suites the policy mandates or lower a threshold below the policy floor.

Release Lifecycle with Evaluation Gate

The diagram below shows the default gate position (before approval) and where evaluation runs fit in the full lifecycle:

An evaluation result can be incomplete when a required test case is skipped by a retryable release-pin conflict during a rollout window. Incomplete suites do not count skipped cases in the pass-rate denominator, but they still fail the release gate until the suite is re-run against a stable active release.

Recurring Quality Monitors

Recurring quality monitors are opt-in, disabled-by-default operator intents that periodically compare a pinned candidate release with a pinned baseline using an exact evaluation-suite version. They provide accountable quality review history; they do not replace Compare or the release evaluation gate. Eligible users configure and act on monitors through the Portal, CLI, or SDK. The AI Dev Kit exposes read-only monitor and review-episode inspection so an assistant can surface due work and evidence without enabling a loop or closing an episode on its own.

Keep these thresholds and verdicts separate:

A complete monitor can pass its own floor while the raw Compare status remains failed. Display the monitor verdict and raw Compare status as distinct values; neither one should be presented as an execution-success or release-gate claim without the corresponding evidence.

Generalized Evaluation Sessions

Release evaluations are gate evidence for a concrete release. Generalized evaluation sessions are advisory comparison runs for asset candidates or asset versions before they become release bindings.

Use /v1/evaluation-sessions to create a session for asset_candidate or asset_version subjects, add manual or imported candidates, and run those candidates against an evaluation_suite. Asset-local and end-to-end session outcomes return gate_eligible: false; they never publish a winning candidate, create an asset version, update a release binding, or clear a release gate by themselves. A human or governed workflow must still promote any selected candidate through the normal asset-version and release lifecycle.

Assistant-generated candidates (source: "assistant_generated") and Orqestra-managed LLM-as-judge runs (judge_mode: "llm_judge") are Enterprise-gated. Community runtimes reject those paths with enterprise_capability_required; deterministic manual/imported candidate runs remain available.

Generated submissions are synchronous and all-or-nothing. Each generated spec produces one candidate and carries a generation_brief; a request may contain at most five generated specs, briefs are capped at 4,000 characters, and a session may contain at most 50 candidates across all sources. Generated create and append requests require a client-minted idempotency_key. Retain the exact payload and key across a timeout, idempotency_request_in_progress, or idempotency_outcome_unavailable; a completed retry returns the original status and body without another dispatch. Rotate the key only after a definitive outcome or when the payload changes.

curl -X POST "http://localhost:8000/v1/evaluation-sessions?workspace_id=$WORKSPACE_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "subject_kind": "asset_candidate",
    "subject_ref": "refund-policy-prompt",
    "idempotency_key": "eval-candidate-7d973ee7",
    "candidates": [{
      "source": "assistant_generated",
      "asset_kind": "prompt_template",
      "generation_brief": "Draft a concise refund-policy prompt with a customer-name variable."
    }]
  }'

Generated candidate summaries return their validated content and governed provenance. Manual and imported summaries return content: null. Append responses additionally return appended_candidates in request order while preserving the bounded whole-session candidates snapshot.

curl -X POST "http://localhost:8000/v1/evaluation-sessions?workspace_id=$WORKSPACE_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "subject_kind": "asset_candidate",
    "subject_ref": "refund-policy-prompt",
    "candidates": [
      {
        "source": "manual",
        "asset_kind": "prompt_template",
        "content": { "template": "Answer using this refund policy: {{policy}}" }
      }
    ]
  }'

curl -X POST "http://localhost:8000/v1/evaluation-sessions/{session_id}/runs?workspace_id=$WORKSPACE_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "candidate_id": "candidate-uuid",
    "suite_slug": "refund-policy-accuracy",
    "execution_mode": "asset_local",
    "judge_mode": "deterministic"
  }'

Generating candidates in the portal

The asset builder’s Test mode (Evaluation Workbench) offers a Generate candidate control alongside the existing manual and import-version candidate flows. It is visible whenever the evaluation-assistant-candidates capability is off — disabled with the capability’s reason — so operators on a Community edition always see the feature and why it is unavailable, rather than a missing button.

When the capability is enabled, clicking Generate candidate opens an inline brief intake: a single free-text description of the desired prompt/skill/agent, submitted as one assistant_generated draft per click. The button’s enabled state, the served asset kinds, the per-request draft cap, and the brief character limit are all read from the capability’s own generation_limits — the portal never hardcodes a client-side kind list or count. Selecting a kind the capability does not serve, or reaching the per-request cap, disables the button with a tooltip naming the served set or the limit.

Clicking Run evaluation submits every draft — manual, imported, and generated — in one POST /v1/evaluation-sessions request. A generated batch mints a client-side idempotency key and retains the exact request body for recovery. The server’s returned content for each generated candidate folds back into the corresponding draft (a positional match against the request’s candidate order), and a candidate whose server-reported candidate_kind is assistant_generated renders a small badge in the candidate table and its result card.

The workbench is deliberately one-shot: it never auto-retries a submission. A 409 idempotency_request_in_progress or 503 idempotency_outcome_unavailable response surfaces a recovery banner with a Retry-After-driven countdown (falling back to 5 seconds without a usable header); the Run evaluation button, the brief intake, and per-candidate removal are locked until the countdown clears. Only an explicit click on Retry resends the exact retained key and payload. Editing the candidate list after the countdown clears — adding, removing, or describing another candidate — abandons the recovery episode and mints a fresh idempotency key for the next submission instead. A 422 generation_needs_clarification response surfaces the server’s clarification question and reopens the brief intake without storing a partial candidate, so the operator can resubmit a more specific brief.

Triggering Evaluations

API

curl -X POST http://localhost:8000/v1/releases/{release_id}/evaluate \
  -H "Content-Type: application/json" \
  -d '{}'

To run specific suites only:

curl -X POST http://localhost:8000/v1/releases/{release_id}/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "suite_slugs": ["refund-policy-accuracy", "tone-safety"]
  }'

Response (202 Accepted):

{
  "evaluation_run_id": "evr-0000-0000-0000-000000000001",
  "release_id": "r1a2b3c4-0000-0000-0000-000000000001",
  "status": "pending",
  "suite_slugs": ["refund-policy-accuracy", "tone-safety"],
  "created_at": "2026-04-11T09:00:00Z"
}

EvaluationResult Status Transitions

Each triggered evaluation run creates one or more EvaluationResult records, one per suite:

pending --> running --> passed
                   --> failed
                   --> error

Viewing Results

Portal

On the release detail page, select the Evaluations tab to see:

• A gate status banner indicating whether the gate is currently passed, failed, or pending (evaluation not yet run or still running).
• A Run Evaluations button that triggers a new evaluation run against all configured suites.
• A results table listing each suite with its pass rate, threshold, and overall status.
• Expandable rows per suite showing per-test-case detail: the input, expected output, actual output, match mode, and individual score.

API — List Evaluation Results

curl http://localhost:8000/v1/releases/{release_id}/evaluations

Response (200 OK):

[
  {
    "evaluation_result_id": "evr-0000-0000-0000-000000000001",
    "release_id": "r1a2b3c4-0000-0000-0000-000000000001",
    "suite_slug": "refund-policy-accuracy",
    "status": "passed",
    "pass_rate": 0.97,
    "threshold": 0.95,
    "test_case_count": 30,
    "test_case_results": [...],
    "created_at": "2026-04-11T09:00:00Z",
    "completed_at": "2026-04-11T09:00:45Z"
  }
]

API — Get a Single Evaluation Result

curl http://localhost:8000/v1/releases/{release_id}/evaluations/{evaluation_result_id}

Gate Enforcement

When a transition is attempted and the gate is not satisfied, the API returns HTTP 422:

{
  "error": "evaluation_gate_failed",
  "message": "Release cannot transition: evaluation gate not cleared.",
  "details": {
    "gate_position": "before_approval",
    "missing_suites": ["tone-safety"],
    "failing_suites": [
      {
        "suite_slug": "refund-policy-accuracy",
        "pass_rate": 0.81,
        "threshold": 0.95,
        "status": "failed"
      }
    ]
  }
}

To clear the gate:

1. Fix the issue in the underlying flow, prompt template, or model configuration.
2. Publish a new asset version and update the release binding.
3. Trigger a new evaluation run (POST /v1/releases/{id}/evaluate).
4. Wait for results to reach passed status.
5. Retry the transition.

Re-running evaluations on the same release bindings (without changing anything) will produce a new result record but is unlikely to change outcomes unless there is an upstream non-determinism you want to average out.

SDK Reference

Python SDK

from orqestra_sdk import OrqestraClient
 
async with OrqestraClient("http://localhost:8000") as client:
    # Trigger evaluation (all configured suites)
    run = await client.releases.evaluate(release_id="r1a2b3c4-...")
 
    # Trigger evaluation (specific suites)
    run = await client.releases.evaluate(
        release_id="r1a2b3c4-...",
        suite_slugs=["refund-policy-accuracy"],
    )
 
    # List all evaluation results for a release
    results = await client.releases.get_evaluations(release_id="r1a2b3c4-...")
 
    # Get a specific evaluation result
    result = await client.releases.get_evaluation(
        release_id="r1a2b3c4-...",
        evaluation_result_id="evr-0000-...",
    )
 
    print(result.status)     # "passed" | "failed" | "error" | "pending" | "running"
    print(result.pass_rate)  # 0.97

CLI Reference

Trigger evaluations:

orqestractl release evaluate <release-id>
 
# Run specific suites only
orqestractl release evaluate <release-id> --suite refund-policy-accuracy --suite tone-safety

List evaluation results:

orqestractl release evaluations <release-id>

Get a specific result:

orqestractl release evaluation <release-id> evr-0000-...

Create and run an advisory asset-candidate evaluation session:

orqestractl evaluation create \
  --subject-kind asset_candidate \
  --subject-ref refund-policy-prompt \
  --candidate-json '{"source":"manual","asset_kind":"prompt_template","content":{"template":"Answer with {{policy}}"}}'
 
orqestractl evaluation sessions --subject-ref refund-policy-prompt
 
orqestractl evaluation run <session-id> \
  --candidate-id <candidate-id> \
  --suite refund-policy-accuracy

orqestractl evaluation is advisory session tooling. It does not activate, publish, bind, or approve release assets. Declarative sync currently keeps evaluation sessions out of manifests; sync continues to rely on explicit release lifecycle commands and the control-plane release gate.

Related

• Releases — State machine, bindings, and approval flow.
• Policies — Policy bundles and how they govern workloads.
• Workloads — Workload configuration including eval_config.
• Assets — How evaluation suites are versioned as governed assets.