Audit & Compliance

Technical reference. This page is for integrations, advanced configuration, or platform operations. For portal tasks, start with the user guide.

Orqestra records a tamper-evident audit trail for every significant action taken on the platform. This guide explains what is recorded, how to verify it, and what to do when something looks wrong.

What’s recorded (always-on baseline)

The governance spine captures the following automatically, with no opt-in required:

• Flow and step lifecycle events — every execution start, step start, step completion, step failure, step retry, execution completion, and subflow launch, attributed to the originating organization, namespace, workspace, workload, flow, and execution.
• Policy evaluations — every obligation evaluation outcome, policy denial, and resource enforcement decision, including the rule that triggered it.
• Operator actions — every config change, workload pause, execution kill, cap override, and circuit-breaker trigger. These are recorded in the hash-chained audit log (governance_audit) in addition to the event stream, giving them stronger integrity guarantees than ordinary events.
• Audit chain integrity checks — the daily verification outcome (v1.lifecycle.audit_chain_verified or v1.enforcement.audit_chain_diverged) is itself a governance event, so the absence of the heartbeat is observable.

What is not recorded by default: prompt content, completion text, tool arguments, memory state, and retrieval results. Content observability is a separate, opt-in capability planned for a future phase.

For the full data model, see the ARCHITECTURE.md Governance Spine section.

Recording a human-oversight event

The human-oversight records describe the oversight posture for an AI system — whether oversight is required, who is assigned, and its readiness. A human-oversight event is a different thing: it records the actual approval, denial, or override act for a specific governed run — who decided, on what channel, when, and any free-text context.

Events are immutable and workspace-scoped. There is no update or delete: once recorded, an event is append-only, and each create appends a hash-chained oversight_event.recorded row to governance_audit (see Two-layer audit model).

Record an event against an existing oversight record with the CLI:

orqestractl human-oversight event record <oversight-id> \
  --execution-id <execution-id> \
  --approver-principal "alice@example.com" \
  --decision approve \
  --channel video_call \
  --occurred-at 2026-07-02T14:30:00Z \
  --context-text "Approved after reviewing the flagged transaction."

List the events for an oversight record (newest first):

orqestractl human-oversight event list <oversight-id>

Both commands operate on the CLI’s configured workspace, like the other human-oversight commands.

Recording an event requires the human_oversight.manage permission; listing requires human_oversight.read. The same routes are available over the SDK (record_event, list_events, get_event) and as POST / GET under /v1/ai/oversight/{oversight_id}/events.

Two fields take values from a fixed, gate-enforced set:

The approver_principal is asserted operator-supplied provenance — a record of who the operator states made the decision — not an authenticated identity.

Auto-issuing a runtime grant from an oversight event

Recording an approve or override event can — optionally, and only when explicitly configured — auto-issue a bounded, expiring runtime permission grant (a policy_exception row, born approved) in the same transaction as the event write. deny never issues, under any configuration. This closes the gap between “a human approved this run” and “the run’s runtime authorization actually materialized”: no second write, no manually re-keyed scope/targets/expiry.

Configuring the issuance policy

The issuance policy lives on the governing oversight posture record, not on the event, and is off by default. Four fields, all additive:

Setting or changing any of the four fields requires the policy.manage permission, in addition to human_oversight.manage. Configuring the policy is a distinct, higher-privileged act from recording an event — a human_oversight.manage holder alone cannot self-authorize a runtime grant end to end.

orqestractl human-oversight update <oversight-id> \
  --grant-issuance-mode on_approve \
  --grant-profile-ref workspace_asset_write \
  --grant-target demo-plugin \
  --grant-ttl-minutes 30

--grant-target is repeatable for multiple targets. The same four options are available on human-oversight create.

What happens when an event is recorded

When the governing posture’s grant_issuance_mode is not disabled and the recorded decision is eligible for that mode, human-oversight event record derives a grant entirely from the posture and the already tenancy-validated execution — never from the event request itself, which cannot carry grant fields. The grant’s scope is always the approved run’s workload, never workspace-wide; profile, targets, and TTL come from the posture; the grant is born approved, with evidence naming the originating event.

The CLI prints the outcome after the normal event JSON:

$ orqestractl human-oversight event record <oversight-id> \
    --execution-id <execution-id> --approver-principal alice@example.com \
    --decision approve --channel video_call --occurred-at 2026-07-02T14:30:00Z
{...event JSON...}
grant issuance: issued (exception_id=...)

Possible outcomes:

The reviewer walk (linkage, both directions)

• From the event — orqestractl human-oversight event list <oversight-id> / event get and the portal’s oversight detail timeline show an issued_grant block (exception_id, status, expires_at) on any event that issued one.
• From the grant — orqestractl policy exceptions get <workspace-id> <exception-id> returns originating_event_id, pointing back to the event that materialized it.
• Both directions are additionally durable in the hash-chained audit log: oversight_event.recorded carries a grant_issuance block on every write where the policy is not disabled; a successful issuance also appends a distinct policy_exception.auto_issued verb (separating delegated issuance from operator-driven two-step approvals at a glance); a denial appends oversight_event.grant_issuance_denied.

Failure semantics

An auto-issued grant is an ordinary policy_exception row afterward: it is visible through the normal exception list/get routes, revocable through the normal revoke route, and swept by the normal expiry worker — auto-revocation on a later event is not implemented; revoke it manually if the approval was a mistake.

Declaring the oversight admission gate

Everything above treats a human-oversight event as pure evidence — recording an event never blocks a run. The oversight admission gate is a separate, opt-in declaration that turns a recorded event into a runtime precondition: once declared and enabled, a high-consequence run (or, for a named workload, every run) is not admitted until a matching approve/override event exists.

Default: off. With no declared policy, nothing changes — every run is admitted exactly as it is today. Declaring the gate is an explicit act on top of the existing oversight posture record; it is never implied by oversight_requirement="required" on that record.

Declaring a policy

A policy is either workspace-wide (applies to every workload in the workspace, but may only gate high_consequence_runs — no blanket blocking of every run in a workspace) or workload-specific (may additionally gate all_runs, unconditionally, for that one workload). The workload-specific row wins when both are declared.

# Workspace-wide: gate only high-consequence runs.
orqestractl human-oversight gating set \
  --oversight-id <oversight-id> \
  --gate-scope high_consequence_runs \
  --freshness-window-seconds 3600 \
  --oversight-wait-seconds 86400 \
  --change-reason "Require sign-off on high-consequence runs this quarter"
 
# Workload-specific: gate every run of one workload.
orqestractl human-oversight gating set \
  --workload-id <workload-id> \
  --oversight-id <oversight-id> \
  --gate-scope all_runs \
  --freshness-window-seconds 3600 \
  --oversight-wait-seconds 86400 \
  --change-reason "Pilot: every run of this workload needs sign-off"
 
# Inspect the declared policy (workspace-wide, or pass --workload-id).
orqestractl human-oversight gating show

--gate-scope accepts high_consequence_runs or all_runs. --freshness-window-seconds bounds how old a matching event may be (1 to 604800, i.e. up to 7 days) — an event recorded earlier than the window never matches. --oversight-wait-seconds (required, 1 to 31536000) is how long a suspended run may wait undecided before it is permanently refused. Declaring or updating a policy requires human_oversight.manage; reading it requires human_oversight.read. There is no delete: disable a gate with --disabled on gating set (the declaration and its audit history stay). The same operations are available over the SDK (HumanOversightClient.get_gating_policy / set_gating_policy / get_workload_gating_policy / set_workload_gating_policy) and as GET/PUT under /v1/ai/oversight/gating-policy[/workloads/{workload_id}]. The portal’s oversight detail page shows a read-only summary of the declaration that names that posture record.

The deny → record → retry loop

Once a policy is declared and enabled, a covered run that has no matching event is denied — fail-closed — with a 403 whose detail carries a machine-readable gap code and the denied execution’s id. That id is exactly what you record the approval against:

1. Run denied. The run is rejected with missing_oversight_event and a denied execution is recorded, e.g. execution_id=<denied-id>.

2. Record the approval against that execution:

   orqestractl human-oversight event record <oversight-id> \
     --execution-id <denied-id> \
     --approver-principal "alice@example.com" \
     --decision approve \
     --channel video_call \
     --occurred-at 2026-07-02T14:30:00Z

3. Retry the run. The retry is a new execution attempt (a new execution id), but it is admitted because it matches on the same workload and the same subject as the denied attempt the event was recorded against — the run does not need to reuse <denied-id>.

4. One event, one run. The event is single-use: a second retry beyond the one it admitted is denied again, with oversight_event_consumed. Record a fresh event per additional run you need to admit.

A recorded deny event is not just “no match” — it actively blocks even a prior approve, until a strictly newer approve/override is recorded (“latest human word wins”). Freshness is measured on the event’s immutable server-recorded time, never the operator-asserted occurred_at.

Every decision (allow and deny) is durably audited (oversight_admission.decided); a deny additionally emits a governance event operators see through the normal notification surfaces. The control-plane oversight-admission service and its API contract are authoritative for the trust boundary and matching semantics.

Decide a protected pending run in Run detail

For new supported native execution and serial acting-call denials, Orqestra stores the exact protected submission and admission lineage before returning the denial. Open that Run detail to review the redacted action and current gap reasons, then use Approve or Deny in its Act panel. An approval creates one fresh governed attempt; the original denied or suspended result remains unchanged, and the panel links to the continuation as it queues, runs, and completes. A stale policy, changed release, revoked execution.run permission, wrong human, or containment fence prevents the continuation.

The CLI provides the same authenticated decision path:

orqestractl human-oversight pending list --execution-id <execution-id>
orqestractl human-oversight pending decide <pending-id> \
  --decision approve \
  --expected-revision <revision>

The command uses the caller’s current human session and submits the exact pending revision. The server independently checks the recent verified session, human_oversight.manage, and the original presenter’s current execution.run authority. SDK users can call HumanOversightClient.list_pending_decisions and decide_pending. MCP can inspect pending state, but cannot make the decision. Historical denials without protected source context remain unavailable; use the legacy event route only to record an act that already happened out of band. Those asserted records remain distinguishable from authenticated inline decisions.

Retention defaults

The platform applies per-class retention policies. Operators can override defaults at the organization, namespace, workspace, or workload level via the governance_retention_overrides table (platform admin permission required).

Execution evidence uses the same retention posture at the hot-row boundary. Evidence payloads are canonicalized, hashed, byte-counted, and grouped by ingest batch before storage; eligible evidence, execution, retrieval, checkpoint, and snapshot rows are archived into deterministic archive records before hot rows are deleted. Backing evidence and checkpoint payload objects are deleted only after the archive transaction commits.

All evidence writes route through a single EvidenceLifecycleService boundary. Every successful write appends a per-artifact governance audit row under the v1.governance.evidence_persisted action; the audit after_state carries the artifact kind, retention class, content hash, and payload size. Unknown retention classes and over-size summary or payload values are rejected before the object-store write. The summary and payload byte caps are configurable via ORQESTRA_EVIDENCE_SUMMARY_MAX_BYTES (default 64 KiB) and ORQESTRA_EVIDENCE_PAYLOAD_MAX_BYTES (default 1 MiB).

Beyond per-artifact caps, evidence ingest enforces a per-organization byte-volume budget so storage growth is bounded by volume, not just artifact count: ORQESTRA_EVIDENCE_WRITE_BYTE_LIMIT_PER_WINDOW (default 1 GiB per window) is checked atomically with the artifact-count limit (ORQESTRA_EVIDENCE_WRITE_RATE_LIMIT_PER_MINUTE), sharing the same window. If either budget would be exceeded the batch is refused before any object-store write (HTTP 429 with a Retry-After), and neither budget is consumed. Admitted and refused byte volumes are exported as orqestra_evidence_ingest_bytes_total and orqestra_evidence_ingest_rejected_bytes_total{reason} so SREs can alert on object-store growth before a bucket or disk fills.

Early disposal is itself an audited action. Any operator-triggered early purge writes an audit row with action=retention.disposal.early and requires a mandatory rationale field, preventing the “destroy evidence via retention manipulation” attack.

Two-layer audit model

Orqestra defends audit integrity in depth. A breach of one layer does not mean tampering goes undetected.

Layer 1 — Hash-chained Postgres. Every row in governance_audit carries a SHA-256 hash of its own canonical form (row_hash) and a pointer to the prior row’s hash (prev_hash). A database trigger raises an exception on any UPDATE or DELETE attempt. The application role used by services has no UPDATE, DELETE, or TRUNCATE permission on this table. Layer 1 is always-on and requires no configuration.

Layer 2 — WORM export to Object Lock storage. When configured, contiguous audit segments are periodically written to the configured external object store. The exporter uses retention resolved from explicit organization policy and an approved, fresh physical-capability snapshot; the retention mode may be GOVERNANCE or COMPLIANCE and is never assumed by this guide. Until the destination, policy, and capability evidence are ready, Layer 2 is unavailable while Layer 1 continues recording.

WORM configuration

To use Layer 2, provision an S3-compatible bucket with Object Lock enabled. The exporter checks the configured destination’s actual capability and refuses to write if it does not satisfy the resolved policy.

Set the destination on the control-plane service. Configure the explicit organization policy and obtain a fresh approved capability observation from Assure → Evidence → WORM audit export before expecting exports to become ready. The observation can also be requested through the authenticated orqestractl audit worm-capability approve command. There is no WORM export enable/disable master flag.

S3 authentication variables (required unless using instance IAM credentials):

The export runs on the worm_export_tick Celery beat task. Export failures emit governance events but never block audit inserts — the application keeps running and recording to Layer 1 even when Layer 2 is degraded.

For local testing, the repository includes a minio-bootstrap compose service (activated by the full or worm profile) that creates orqestra-audit-worm with Object Lock automatically. See docs/process/DEVELOPMENT.md for details.

Browsing export metadata and storage usage

Operators can inspect WORM export state and object-store growth without direct MinIO/S3 access. Every successful export records a durable, metadata-only receipt (object key/version, segment hash, audit-id range, byte size, retention) that backs both a CLI and a platform-scoped HTTP API:

• orqestractl audit worm-status — exporter health: high-water cursor, audit head, lag, consecutive failures.
• orqestractl audit worm-segments [--limit N] — segment receipt list, newest first, metadata only.
• orqestractl audit worm-storage — object-store byte usage: total WORM exported bytes and segment count, plus live (non-archived) evidence bytes by retention class. Archived evidence is not counted (the archive index records no byte size); WORM exported bytes are counted in full.

The same data is exposed over GET /v1/governance/worm-export/{status,segments,segments/{id},storage-summary} and a bounded POST .../segments/{id}/verify, all requiring the platform governance.worm_export.view permission. storage-summary accepts an optional organization_id for a per-org evidence breakdown. None of these stream segment bodies or expose object-store credentials.

Platform administrators with governance.worm_export.manage can approve a jurisdiction mapping and request a fresh observation at POST /v1/governance/worm-export/capabilities/approve-and-observe. Supply the approved jurisdiction, mapping_version, source_ref, and authority_ref; the server supplies physical storage facts, approver identity, timestamps, and the durable snapshot. The Portal exposes this form on the WORM export page and the authenticated CLI command is orqestractl audit worm-capability approve.

Platform operators also get a portal page at Assure → Evidence → WORM audit export (/operator/worm-export). It is metadata-first and consumes the same API:

• Exporter health cards: enabled state, destination, lag (audit head minus high-water), consecutive failures vs threshold, and last successful export.
• Storage overview cards: WORM exported bytes and segment count, plus live (non-archived) evidence bytes broken down by retention class.
• Segment table: audit-id range, row count, size, retention, exported time, and a truncated hash; paginate older history with Load more.
• Segment detail (/operator/worm-export/{id}): full receipt metadata with copyable object key and SHA-256, plus a Verify segment action that runs the bounded re-hash check and reports intact / mismatch. It never downloads the segment body to the browser, and reports “verification unavailable” (rather than a false pass) when the object store is unconfigured.

The page is hidden from anyone lacking governance.worm_export.view, and renders explicit loading, empty, permission-denied, and not-found states. WORM export metadata is platform-global, so the page carries no tenant or workspace selector — it is not workspace-scoped.

Using orqestractl audit verify

The verify command replays the governance_audit hash chain and reports any divergence. It connects directly to Postgres; no running control-plane service is needed.

Syntax

orqestractl audit verify [--from N] [--to M] [--worm]

• --from N — Start from audit row N (default: 1, the genesis anchor).
• --to M — Stop at audit row M (default: current chain head).
• --worm — Also download and verify WORM segments from ORQESTRA_WORM_EXPORT_DESTINATION.

Exit codes

When to run it

• On-demand — after a DB restore, after an incident, or any time you want an out-of-band integrity check.
• Scheduled — the audit_verify_job runs automatically at 03:00 UTC daily and emits a governance event. The CLI command gives you the same check interactively.

Sovereignty evidence packs

For releases that pin a sovereignty_profile asset, operators can inspect and export a release-scoped JSON evidence pack from every operator surface:

curl -H "Authorization: Bearer $ORQESTRA_API_KEY" \
  "$ORQESTRA_CONTROL_PLANE_URL/v1/releases/<release-id>/sovereignty-evidence-pack"

• Portal: open the release detail page and use the Sovereignty tab to review the active profile, provider route decisions, exceptions, and JSON export availability.
• CLI: run orqestractl release sovereignty-evidence-pack <release-id>.
• SDK: call await client.releases.get_sovereignty_evidence_pack("release-id").
• MCP: call get_sovereignty_evidence_pack(release_id="release-id").

The pack includes the applied profile, provider and region route decisions from the bound model_config, execution evidence artifacts, retrieval references, approval records, related governance audit rows, exception basis, key-control posture, operator-control posture, retention policy, and export policy. The data_movement_decisions and operator_access_events fields are reserved, schema-stable action buckets and remain empty until their matching audit producers ship. The endpoint requires release.view on the release workspace and only returns JSON when the pinned profile allows json in export.export_formats.

How to interpret the events

v1.lifecycle.audit_chain_verified (severity: info) — The scheduled verifier completed a clean pass. The event attributes include rows_verified, head_audit_id, head_row_hash_hex, and duration_ms. The presence of this event at the expected cadence is itself a positive signal.

v1.enforcement.audit_chain_diverged (severity: critical) — The verifier detected a hash mismatch. The event attributes include rows_scanned, first_divergence_audit_id, and a one-line diagnostic describing the type of mismatch (row-hash or link-hash). Treat this event as a P1 security incident.

What divergence means and how to respond

A v1.enforcement.audit_chain_diverged event, or an orqestractl audit verify exit code of 1, means the hash chain is broken at a specific row. This is a security incident, not a software bug.

Immediate response:

1. Page your security team. Treat this as a P1 incident.
2. Note the first_divergence_audit_id from the event attributes or the CLI output. This is the earliest row where the tamper is detectable — the actual modification may have occurred at or before this row.
3. Do not truncate or repair the governance_audit table. Modifying the table destroys the forensic record. The table’s immutability triggers will block most attempts, but a superuser could bypass them. Make this a standing operational policy.
4. If Layer 2 (WORM) is enabled, run orqestractl audit verify --worm to check whether the WORM segments are still clean. A clean WORM result alongside a diverged Layer 1 narrows the window in which tampering occurred (between the last clean WORM segment export and now).
5. Investigate application-layer access logs and Postgres superuser audit logs for unauthorized writes to governance_audit in the window around first_divergence_audit_id.
6. After investigation, re-run orqestractl audit verify on a clean environment (DB restore from a pre-divergence backup, if available) to confirm the chain was intact before the incident.

Escalation paths:

Policy version change-impact (blast radius)

Before retiring a range of a policy bundle’s versions, operators can inspect the change-impact — every governed decision that ran under the versions being superseded:

curl -H "Authorization: Bearer $ORQESTRA_API_KEY" \
  "$ORQESTRA_CONTROL_PLANE_URL/v1/policies/<bundle-id>/version-impact?from_version=<v>&to_version=<v>&workspace_id=<ws>"

The from_version and to_version query params define the half-open [from_version, to_version) set of the bundle’s versions being retired — from_version is included, to_version is not. Both must belong to the bundle, and from_version must strictly precede to_version in the bundle’s version history. The optional workspace_id scopes the read to a single workspace.

The response reports:

• rows — one entry per impacted decision (workload, release, scenario, outcome, reason codes, governing version label, and a representative execution).
• metrics — a rollup over those decisions (total_decisions, distinct_workloads, distinct_releases, and counts by_outcome / by_decision_kind).
• version_unknown — a separate band counting decisions for the bundle that carry no resolved version. These are surfaced explicitly (their own count) rather than folded into any version range, so historical decisions written before version capture are never mis-attributed to a range you are retiring.

Consume it from every operator surface:

• CLI: orqestractl policy version-impact <bundle-id> --from-version <v> --to-version <v> [--workspace-id <ws>].
• SDK: await client.policy_simulation.get_version_impact("bundle-id", "from-version", "to-version", workspace_id="ws").

The endpoint requires POLICY_VIEW on the bundle’s workspace. It returns 404 if the bundle does not exist, is not a policy bundle, or is not visible in the requested workspace, and 422 if the version range is inverted or empty.

Further reading

• ARCHITECTURE.md — Governance Spine
• ADR 0057 — Hash-Chained Audit + Two-Layer WORM 
• CLI Reference — orqestractl audit verify