Execution Recovery & Control

Technical reference. This page is for integrations, advanced configuration, or platform operations. For portal tasks, start with the user guide.

Execution recovery covers the operational actions used after a workflow is running or has already finished: pause, resume, inspect a checkpoint snapshot, rerun from a checkpoint, replay from the beginning, and backfill missed trigger windows.

These controls are governance-sensitive. They create or affect real executions, enforce workspace permissions, and preserve actor and recovery metadata so the action remains auditable.

When To Use Each Operation

Permissions

Internal runtime checkpoint routes are service-only. Users should not call POST /v1/executions/{execution_id}/checkpoints, PATCH /v1/executions/{execution_id}/checkpoints/{checkpoint_id}, or GET /v1/executions/{execution_id}/checkpoints/latest directly.

Pause And Resume

Pause and resume use Redis execution-control signals. Pause does not forcibly terminate the process at an arbitrary instruction; the runtime cooperates with the signal and stops at supported checkpoints.

API

curl -X POST http://localhost:8000/v1/executions/$EXECUTION_ID/pause
curl -X POST http://localhost:8000/v1/executions/$EXECUTION_ID/resume

Pause response:

{
  "execution_id": "f7f0ad4c-0000-0000-0000-000000000001",
  "status": "pause_signal_sent"
}

Resume response:

{
  "execution_id": "f7f0ad4c-0000-0000-0000-000000000001",
  "status": "resume_signal_sent"
}

CLI

orqestractl execution pause $EXECUTION_ID
orqestractl execution resume $EXECUTION_ID

Python SDK

from orqestra_sdk import OrqestraClient
 
async with OrqestraClient("http://localhost:8000") as client:
    await client.executions.pause(execution_id)
    await client.executions.resume(execution_id)

Snapshot

Snapshot returns the latest checkpoint metadata visible to the caller. Use it when deciding whether a rerun can resume from the expected step.

curl http://localhost:8000/v1/executions/$EXECUTION_ID/snapshot
orqestractl execution snapshot $EXECUTION_ID

Example response fields:

{
  "execution_id": "f7f0ad4c-0000-0000-0000-000000000001",
  "current_step_id": "retrieve_context",
  "previous_step_id": "load_request",
  "status": "paused",
  "pause_reason": "operator_pause",
  "workspace_id": "8a0b0000-0000-0000-0000-000000000001"
}

Rerun From A Step

Rerun forks a new execution from a checkpoint. The original execution must be failed, killed, or completed. If from_step_id is omitted, Orqestra uses the latest recorded checkpoint step.

curl -X POST http://localhost:8000/v1/executions/$EXECUTION_ID/rerun \
  -H "Content-Type: application/json" \
  -d '{
    "flow_slug": "support-triage-flow",
    "from_step_id": "classify_ticket",
    "environment": "production"
  }'

Operational notes:

• The new execution carries _recovery.type = "rerun" and _recovery.original_execution_id in its inputs.
• The actor is the caller. Supplying a different actor_ref is rejected as impersonation.
• If environment is omitted, it is resolved from the source execution’s historical release. An explicit value must match that environment; a mismatch returns recovery_environment_mismatch.
• Recovery requires a trusted checkpoint identity and integrity envelope. A missing, mixed, or unverifiable envelope fails closed with checkpoint_integrity_invalid before hydration or dispatch.
• Tool posture is resolved against the release that will actually run, not blindly copied from the old execution.
• Before dispatch, Orqestra evaluates checkpoint recovery_posture and records a governed execution.recovery_disposition audit event. Committed effects outside internal runtime state require idempotency or reconciled sealed evidence before automated recovery proceeds.
• For a retry-safe request, send a lowercase RFC-4122 client_request_id. The server reserves the target execution ID before dispatch; the same key and historical inputs return the original target as replayed, while reusing the key with different inputs returns recovery_idempotency_conflict. Omitting the key preserves legacy per-call behavior.
• Signed checkpoints are admitted only when the workspace Vault keyring contains the referenced, currently unrevoked Ed25519 key and the signature is valid for the exact checkpoint identity. Key rotation uses Vault KV version CAS; a revoked key fails closed even during an overlap window.

Replay From The Beginning

Replay starts the flow from the beginning. environment is optional: when omitted, replay resolves it from the ORIGINAL execution’s own release and re-runs in that SAME environment — never a hardcoded default, so replaying a staging run can never silently dispatch a live production run. Pass environment explicitly only when it matches the source environment; a mismatch returns recovery_environment_mismatch. The value is validated against the org’s governed environment ladder. Replay uses the current active release for the resolved environment and pins that release through dispatch. Set freeze_release when the replay should carry a frozen-input hint in recovery metadata; it does not switch execution back to the original release.

curl -X POST http://localhost:8000/v1/executions/$EXECUTION_ID/replay \
  -H "Content-Type: application/json" \
  -d '{
    "flow_slug": "support-triage-flow",
    "freeze_release": true
  }'

Use replay for investigation and controlled reproduction. Use rerun when the known-good checkpoint state is the safer starting point.

Recovery Admissibility

Recovery decisions use a closed disposition set:

Checkpoint posture may classify committed side effects as internal_state, model_provider, retrieval, external_http, tool_side_effect, memory_write, evidence_write, artifact_write, human_approval, runtime_limit, or policy_obligation. Internal-only state does not require external evidence. Other committed classes require an idempotency key plus a clean sealed external-runtime evidence summary and matching reconciliation event. Missing or unsealed evidence returns operator_review_required; contradictory evidence returns recovery_denied.

Checkpoint writers declare effects under recovery_posture.side_effects with committed: true, or under recovery_posture.committed_side_effects when every entry is already committed. Each committed external entry must include a class, step_id, and idempotency_key.

Trigger Backfill

Backfill computes missed cron ticks over a time window and dispatches one real execution per tick. Use dry_run: true first to inspect the tick count before dispatching.

curl -X POST http://localhost:8000/v1/triggers/backfill \
  -H "Content-Type: application/json" \
  -d '{
    "trigger_binding_id": "1a8e0000-0000-0000-0000-000000000001",
    "flow_slug": "daily-summary-flow",
    "workload_id": "7b870000-0000-0000-0000-000000000001",
    "cron": "0 8 * * *",
    "timezone": "UTC",
    "from_time": "2026-05-20T08:00:00Z",
    "to_time": "2026-05-22T08:00:00Z",
    "max_concurrent": 3,
    "dry_run": true,
    "environment": "production"
  }'

Dry-run response:

{
  "trigger_binding_id": "1a8e0000-0000-0000-0000-000000000001",
  "window": {
    "from": "2026-05-20T08:00:00+00:00",
    "to": "2026-05-22T08:00:00+00:00"
  },
  "total_ticks": 3,
  "dispatched": 0,
  "failed": 0,
  "dry_run": true,
  "ticks": [{ "tick_time": "2026-05-20T08:00:00+00:00", "status": "pending" }]
}

When dry_run is false, each tick result reports dispatched with an execution_id or failed with an error.

Investigation Checklist

1. Open the execution detail and confirm status, release, workload, and workspace.
2. Read the latest snapshot before rerun or replay.
3. Check the release state and bound policy bundle for the target environment.
4. For rerun, choose an explicit from_step_id when the latest checkpoint is not the intended restart point.
5. For replay, decide whether freeze_release is required as investigation metadata for frozen-input handling.
6. For backfill, run a dry run and verify total_ticks before dispatch.
7. After recovery dispatch, inspect the new execution’s evidence graph.

Failure Modes

Related Guides

• Executions in CLI Reference
• Triggers
• Evidence Graph
• Audit & Compliance