Compare

Technical reference. This page is for integrations, advanced configuration, or platform operations. For portal tasks, start with the user guide.

Compare runs a governed, N-cohort experiment: a candidate configuration against one or more baselines, over a shared dataset, scored on the same metrics. It answers “which of these should I ship?” with a per-metric scorecard, significance/confidence stats, a failure breakdown, and an audit-grade report with a follow-up-gated sign-off.

Compare lives at Evaluation → Compare in the portal, alongside Monitor (the read-only longitudinal quality view — see the design spec for how the two modes divide the Evaluation surface).

Concepts

Comparison run (ExperimentComparisonRun) — one Compare execution: a frozen dataset version plus an ordered list of cohorts. Status moves setup → running → done | failed | cancelled.
Cohort — one arm of the comparison. Cohort 0 is always the candidate; every other cohort is a baseline. A cohort is either:
- a release cohort ({"cohort_kind": "release", "release_id": ...}) — an existing release, evaluated as-is, or
- a variant cohort ({"cohort_kind": "variant", "base_release_id": ..., "overrides": {"<role>": "<asset_version_id>"}}) — a base release with one or more bindings swapped for a candidate asset version (e.g. a new prompt template). A variant is stored as plain data; it never materializes a release.
Dataset — an evaluation_suite governed asset. Its version is frozen at run creation, so every cohort is scored against the exact same cases even if the dataset is edited later.

Setting up a comparison

In the portal, the setup panel has three fields: Comparing (add 2+ release or variant cohorts via the cohort picker), On dataset (pick an evaluation_suite asset from the dataset picker), and Budget cap (optional — see Budget cap). The Run button is disabled until at least two cohorts and a dataset are selected.

Equivalently, over the API:


# List datasets available in the workspace
curl "http://localhost:8000/v1/evaluation-datasets?workspace_id=$WORKSPACE_ID" \
  -H "Authorization: Bearer $TOKEN"
 
# Create the run (candidate first — cohort_index 0)
curl -X POST "http://localhost:8000/v1/experiment-comparisons?workspace_id=$WORKSPACE_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_asset_id": "<evaluation_suite-asset-id>",
    "cohorts": [
      { "cohort_kind": "release", "release_id": "<candidate-release-id>" },
      { "cohort_kind": "release", "release_id": "<baseline-release-id>" }
    ],
    "budget_cap_cents": 500
  }'

A variant cohort instead of a second release:


{
  "cohort_kind": "variant",
  "base_release_id": "<baseline-release-id>",
  "overrides": { "prompt_template": "<candidate-asset-version-id>" },
  "label": "new-refund-prompt"
}

Each override is validated the same way a release binding is — the asset version must resolve inside the workspace (or a global built-in) and match the role’s expected asset kind — so a variant cannot smuggle in an unapproved or wrong-kind asset. The run is created in status: "setup"; nothing is dispatched yet.

Reliability and recovery

Compare records bounded, fixed error codes rather than provider, model, document, prompt, output, credential, or raw exception text. A worker admits a case in a private checkpoint before the external runtime call and closes that database transaction before awaiting the call. If cancellation wins while an admitted call is finishing, the run stays cancelled but the matching private checkpoint records the factual completion, cost, score, counter, and runtime evidence. The cancelled cohort is not published as a partial public outcome.

The public response filters the private checkpoint. An interrupted run exposes only its terminal status and factual counters; its cases endpoint has no rows or execution links for the incomplete cohort. A separate fully completed run is the source of public per-case execution links in Case Debug. Re-run creates a new comparison rather than replaying a cancelled run.

If a worker is redelivered with an admitted or completed orphan checkpoint, it fails with comparison_dispatch_outcome_ambiguous and does not replay the provider call. Only cohorts whose existing outcomes committed can be resumed, and their recorded costs/counters are restored once. A database hard-kill during a provider call remains an ambiguity residual; the durable checkpoint and evidence are retained for diagnosis. The optional budget cap is post-hoc: spend is known only after the runtime execution record exists, so dispatch stops subsequent cases/cohorts once the recorded total reaches the cap.

CLI:


orqestractl experiment datasets --workspace-id $WORKSPACE_ID
orqestractl experiment create \
  --workspace-id $WORKSPACE_ID \
  --dataset-asset-id <evaluation_suite-asset-id> \
  --cohort-json '{"cohort_kind":"release","release_id":"<candidate-release-id>"}' \
  --cohort-json '{"cohort_kind":"release","release_id":"<baseline-release-id>"}'

Budget cap

budget_cap_cents is an optional, per-run soft spend cap in cents, distinct from the durable per-experiment daily budget (the Experiment budgets panel on the same page, which configures workspace/experiment daily_budget_cents independently of any one comparison run). Once set, dispatch stops once the run’s accumulated spend reaches the cap. The run response’s spent_cents_so_far / budget_cap_cents back the live spend meter shown while a run is running or done. There is no CLI/SDK option for the cap today — set it via the portal or the raw POST /v1/experiment-comparisons call above.

Running it

Once a run exists, Start run dispatches every cohort’s test cases against the frozen dataset:


curl -X POST "http://localhost:8000/v1/experiment-comparisons/$RUN_ID/start?workspace_id=$WORKSPACE_ID" \
  -H "Content-Type: application/json" \
  -d '{"background": false}'

background: false (the default) blocks the request until the run reaches a terminal state (done/failed); background: true enqueues the dispatch and returns as soon as the run flips to running — poll GET /v1/experiment-comparisons/{run_id} for progress. Only a setup run can be started (409 otherwise), and a run in setup or running can be cancelled:


orqestractl experiment run --workspace-id $WORKSPACE_ID --run-id $RUN_ID --background
orqestractl experiment status --workspace-id $WORKSPACE_ID --run-id $RUN_ID
orqestractl experiment cancel --workspace-id $WORKSPACE_ID --run-id $RUN_ID

The portal always dispatches in the background and shows live progress while running: case count/percent, a chip per cohort (done/running/queued), and a time-left estimate, alongside Cancel and Run in background (which collapses the detail view — the run itself is already running in the background; this only hides the progress bar/chips, Cancel stays available). Once done, the header shows a Last run relative timestamp (e.g. “2h ago”) and Re-run, which starts a fresh comparison from the same dataset, cohorts, and budget cap.

Reading the results

Once a run reaches done, GET /v1/experiment-comparisons/{run_id} returns every cohort’s pass_rate plus a scorecard:

Metrics — one row per scored metric, with each cohort’s value and an is_winner flag (ties can mark more than one cohort as a winner, and then cast no vote toward the overall winner).
Significance — a two-proportion z-test of each baseline’s pass rate against the candidate’s (p_value, significant); null until both sides have recorded an outcome.
Confidence — one entry per cohort: a Wilson-score pass-rate confidence interval, the sample-adequacy verdict, and the minimum detectable effect (MDE) its sample size can resolve at 80% power / alpha 0.05.

The portal renders these as the metric scorecard, a confidence-interval forest plot, and a sample-adequacy strip. Below that, Failure analysis classifies every case candidate-vs-baseline as Improved (the candidate passed, a baseline didn’t), Regressed (the candidate failed, a baseline passed — the ship gate), Both wrong (candidate and every baseline failed), or Agree (no divergence), behind Regressions / Improvements / All diverged / All tabs (Regressions shown by default). Every case renders inline as a full-width, per-cohort pass/score breakdown — the divergent cohort tinted — and Debug → opens the case-debug drawer with its per-cohort execution trace and evidence:


curl "http://localhost:8000/v1/experiment-comparisons/$RUN_ID/cases?workspace_id=$WORKSPACE_ID"

Report and sign-off

Generate report freezes a new, versioned snapshot of the run’s summary, scorecard, and failure analysis — regenerating never edits a prior version, so what a reviewer signed off on stays exactly what they saw:


curl -X POST "http://localhost:8000/v1/experiment-comparisons/$RUN_ID/report?workspace_id=$WORKSPACE_ID"

Every regression the candidate has against a baseline becomes an open follow-up item on the report. Signing off is gated on those follow-ups:


curl -X POST "http://localhost:8000/v1/experiment-comparisons/reports/$REPORT_ID/sign-off?workspace_id=$WORKSPACE_ID" \
  -H "Content-Type: application/json" \
  -d '{"decision": "approve"}'

Any unresolved follow-up plus no override returns 409 with the open-items list.
override: true against open items requires override_justification.
decision is one of approve (verdict promote), hold (verdict hold), or request_changes (verdict inconclusive); it is written to the run’s verdict and audited. Sign-off records a decision — it does not transition a release on its own. Promoting the winning release to active is the normal release transition flow.