Test and evaluate changes

Use evaluation to assess a release against test cases and compare alternatives before making a release decision. Results describe the tested cases and configuration; they do not guarantee all future behavior.

Before you start

Select the workspace containing the release and evaluation data. You need permission to view releases and metrics; running evaluations or comparisons requires evaluation-management access. Creating an evaluation dataset requires asset authoring access.

Prepare representative, approved test data and agree on the quality criteria. Starting evaluations, workflow tests, or comparison runs can execute workloads, call connected services, and incur costs. Check the environment, connections, and budget before you run them.

Run evaluations for a release

1. Open Assure → Releases and select the release.
2. Confirm the workload, environment, and bound asset versions.
3. Open Evaluations and select Run Evaluations.
4. Wait for results, then review Suite, Status, Pass Rate, and Created.

Check that the suite and result belong to the candidate you intend to assess. A missing result or a dash is not a passing result. If no appropriate suite is configured, ask the workload owner to prepare it before treating evaluation as complete.

Monitor existing results

Open Build → Evaluation → Monitor to inspect existing suite trends, regressions, and failing cases. Browsing Monitor does not start runs. Select available result details to investigate the cases behind a summary.

No evaluations yet means the selected view has no recorded results in its window; it does not establish that the workload is safe or broken. Run the intended release evaluation only when you are ready for execution and its costs.

Compare alternatives

A comparison runs the same dataset against multiple cohorts: selected releases or variants of a release. A variant changes selected configuration for comparison without itself activating a release.

1. Open Build → Evaluation → Compare.
2. Select Add cohort and choose at least two cohorts. Confirm which release or variant each represents.
3. Choose the evaluation dataset. If none exists, use Create an evaluation dataset when offered, or ask an authorized builder to add one.
4. Set Budget cap (USD) if you need a run-specific spend limit. Leaving it blank provides no comparison-specific cap; other organization limits still apply.
5. Review the cohorts, dataset, environment context, and expected external effects before selecting Run. This creates the comparison setup.
6. In the new run panel, select Start run to execute it. Watch progress and recorded spend. Run in background minimizes the panel while work continues; Cancel requests cancellation.
7. Review the final status, scorecard, sample adequacy, uncertainty, and failure analysis before drawing a conclusion.

A budget cap stops further dispatch when reached. Do not assume cancellation reverses completed calls or removes costs already incurred. Re-run creates and starts another comparison with the stored setup, so review its costs and effects before selecting it.

Record the assessment

When a comparison is ready for review, select Generate report. Regenerate report creates another report version rather than editing the earlier one. Review the report and its follow-ups before using Approve, Hold, or Request changes, then Record sign-off when your role and the report allow it.

Open follow-ups block sign-off unless you explicitly choose Override and sign off anyway and provide the required justification. Use that option only within your organization’s decision authority, and confirm the recorded decision and override in the report. A comparison recommendation or report sign-off does not activate a release; release approval and activation remain separate actions.

Set up recurring quality checks

Authorized evaluation managers can open Quality to configure recurring checks. These produce quality findings and accountable review episodes; they do not themselves gate release activation or runtime execution.

1. Select New monitor. Choose the workload, environment, candidate and baseline releases, evaluation suite, and suite version.
2. Review the check interval, time limits, minimum pass rate, budget in cents, and optional backup reviewer. Select Create monitor; new monitors start disabled.
3. Inspect the saved configuration. To opt in, select Edit configuration, choose Enable recurring checks, and Save changes. Enabling it authorizes recurring executions and their costs. Clear that choice and save to disable future recurring checks.
4. Review the monitor’s occurrences and review episodes. Recheck now requests another check and can consume resources.
5. If assigned a review episode, enter an Acknowledgement reason and select Acknowledge episode. Acknowledgement does not close it. Resolve the issue and obtain a qualifying later passed recheck before entering a Close reason and selecting Close episode.

Check the saved state after each action. If configuration changed concurrently or closing is refused, refresh the monitor and episode, read the reason, and resolve the remaining requirement rather than repeating the same request.

If you cannot proceed

Next

Investigate execution evidence, review a release, or inspect resource limits.