Observability & Cost Attribution

Technical reference. This page is for integrations, advanced configuration, or platform operations. For portal tasks, start with the user guide.

Orqestra currently provides four observability surfaces:

1. structured JSON logs
2. OpenTelemetry trace export hooks
3. Prometheus metrics
4. persisted execution telemetry and cost summaries

Structured Logs

Runtime and control-plane services emit structured JSON logs with request, trace, and bounded Orqestra correlation. A typical log record includes:

{
  "timestamp": "2026-04-10T14:32:01Z",
  "level": "info",
  "service": "runtime-gateway",
  "message": "step_completed",
  "request_id": "req-123",
  "traceparent": "00-...",
  "tenant_id": "tenant-123",
  "workspace_id": "workspace-123",
  "workload_id": "support-router",
  "execution_id": "exec-123",
  "release_id": "release-123",
  "step_id": "classify",
  "step_type": "llm_call",
  "duration_ms": 1230
}

Traces

The runtime emits spans for flow execution and step execution. Current span attributes are centered on step identity and runtime correlation rather than a large custom tracing schema.

The important operator takeaway is:

• each flow execution can be correlated through request and trace context
• trusted Orqestra IDs join logs and spans across internal service calls
• each step produces traceable execution timing
• traces align with logs and Prometheus metrics
• portal proxy logs record upstream paths without query parameters

Metrics

Both the control plane and runtime expose a /metrics endpoint. Current metric coverage includes:

Workflow-duration outcome labels are bounded to accepted executor terminals: success, error, blocked, budget_exceeded, killed, paused, approval_required, denied, and concurrency_lease_lost.

Local development can pair this with Jaeger, Prometheus, and Grafana using the observability profile described in the repository README.

Flow-Level Observability Settings

The public flow schema currently supports these top-level observability fields:

Example:

name: customer-support-router
description: Route customer queries to the right workflow.
trigger:
  type: api_request
  inputs_schema:
    type: object
    required: [question]
    properties:
      question:
        type: string
observability:
  trace_all: true
  trace_content: false
  metrics: [latency, token_count, cost]
  context_snapshots: true
  max_context_size: 512
entry_step: classify
steps:
  - id: classify
    type: llm_call
    inputs:
      messages:
        - role: user
          content: "{{trigger.question}}"

Individual steps can also declare trace, trace_content, and metrics.

Execution Telemetry

Completed executions persist a telemetry_summary with aggregate and per-step LLM token data.

Token counts in the telemetry summary reflect actual usage reported by the model provider API, not pre-call estimates. The runtime clamps each step’s max_tokens to the remaining budget before the call is made, so token budgets are consumed accurately and premature budget exhaustion caused by over-estimation is avoided. When a provider omits total_tokens or supplies a value that normalizes as invalid or zero, the runtime uses prompt_tokens + completion_tokens consistently for budget debit, step output, and persisted telemetry.

Example:

{
  "totals": {
    "llm_step_count": 2,
    "priced_step_count": 2,
    "prompt_tokens": 1024,
    "completion_tokens": 256,
    "total_tokens": 1280,
    "estimated_cost_usd": 0.0192,
    "cost_schema_version": 1,
    "cost_status": "complete",
    "cost_sources": ["governed_commit"],
    "cost_source_subtotals": { "governed_commit": 0.0192 },
    "cost_attempt_coverage": {
      "attempts_complete": 2,
      "attempts_partial": 0,
      "attempts_unrecorded": 0,
      "accounting_omissions": 0,
      "presentation_omissions": 0
    }
  },
  "steps": [
    {
      "step_id": "classify",
      "subexecution_id": "run-local-scope-01",
      "step_type": "llm_call",
      "model": "your-provider/your-model",
      "duration_seconds": 0.84,
      "prompt_tokens": 512,
      "completion_tokens": 128,
      "total_tokens": 640,
      "cost_per_input_token": 0.000005,
      "cost_per_output_token": 0.000015
    }
  ]
}

The totals carry the canonical v1 cost-evidence block. cost_schema_version tags the encoding; cost_status records the population’s reconciled coverage (complete, partial, or not_recorded); cost_sources, cost_source_subtotals, and cost_attempt_coverage expose the source set and per-status coverage behind the amount. estimated_cost_usd is the KNOWN SUBTOTAL of observed executions — an observed lower bound, never a full-day or full-window claim — and serializes as null when the population carries no known cost. Unknown cost is never substituted with zero: a null amount paired with cost_status: "not_recorded" means no known cost evidence, not free spend.

Read the evidence state before using a cost amount:

• complete means every member in the reported population has canonical, versioned cost evidence.
• partial means the amount is the known subtotal of the members with usable evidence; cost_sources and cost_source_subtotals identify what is included, while cost_attempt_coverage shows complete, partial, and unrecorded members.
• not_recorded means no member contributes a known amount, so the amount is null.

Cost comparisons, rankings, and deltas must be suppressed when populations are mixed, partial, legacy, or otherwise ineligible. Use the returned comparability reason and coverage rather than treating a missing amount as zero or comparing known subtotals as if they were full-population totals.

subexecution_id is an optional, opaque run-local discriminator. On GET /v1/executions/{execution_id} and GET /v1/executions/{execution_id}/cost, root-flow rows serialize it as null. The stored execution_cost_summary artifact omits the field only for root-flow rows and preserves it on merged child-execution rows, such as subflow or subagent steps, where it carries a string value. Consumers must treat null and absence identically as root-flow and correlate a row by (subexecution_id, step_id) instead of the raw step id alone. The same root rule applies to execution_trace.v2: producers omit the field, while readers accept an explicit null as the root scope.

Per-step duration_seconds is a finite numeric measurement when present. Persisted missing, null, boolean, string, or non-finite values serialize as null on execution list, detail, cost, and analysis reads.

The aggregate totals object is authoritative for run-wide governance. Runtime aggregation continues across every model-bearing occurrence even after the bounded steps array reaches 10,000 rows. Readers drop malformed or unsafe scoped step rows instead of erasing their scope and misattributing them to the root flow. Producers and readers retain at most 10,000 telemetry step rows, while the semantic trace retains at most 500 spans; the authoritative totals are not reduced by those attribution caps. Consequently, per-step token, cost, or span sums can be lower than the corresponding run total; consumers must not recompute run totals from the step array. In that case execution analysis reports model_usage, cost_usage, and affected child attribution as partial, even though the aggregate remains usable. Repeated attempts consume telemetry by scoped step occurrence, so a failed attempt without provider usage cannot take the successful retry’s row. When a child execution was attempted but produced no attributable child step evidence, execution analysis reports the subexecution evidence family as partial, not complete or not_required; one evidenced sibling cannot upgrade that incomplete attempt.

execution_trace.v2 is schema-authoritative. Readers do not fall back to a legacy steps array when a v2 trace is present. Span identifiers, timestamps, duration, metrics, failure/redaction markers, retention values, source refs, and evidence-family rows are bounded and validated strictly. A malformed schema-present field, duplicate or malformed family row, or reader-side span rejection makes step_trace partial instead of being silently discarded as if the trace were complete. The trace execution id and outcome must match the owning execution. The producer summary is required, and its exact key set, schema version, span count, omitted count, outcome, failure kind, and evidence status must agree with the payload and owner. The required evidence_families array contains at most 32 canonical, uniquely named rows. Trace and telemetry integer fields use the signed 64-bit domain; booleans are not accepted as summary integers. Timeline reads return no rows for a null/malformed v2 envelope, a missing summary or family array, duplicate/noncanonical families, an invalid span, or more than 500 stored spans.

The 500-span bound applies once to the complete semantic trace. The 500th span is retained. summary.omitted_span_count includes both spans rejected during producer validation and spans beyond the bound; any omission makes the overall evidence status partial and downgrades every affected evidence family. This includes model/cost, tool, retrieval, guardrail, approval, context, admission/release, and child-execution evidence as applicable. An omitted child step therefore makes both step_trace and subexecution partial. An omitted generic span still makes the overall trace partial even when it has no dedicated family.

Persisted Telemetry Safety

Control-plane telemetry reads treat persisted JSON as hostile input. A raw JSON document containing an escaped NUL or an unpaired surrogate is rejected before PostgreSQL extraction can fail; escaped literal text is not confused with a decoded surrogate. Numeric validation remains field-local, so an unrelated extreme exponent cannot erase valid sibling metrics. PostgreSQL-native and fallback readers share binary64 rounding, signed-zero/underflow, signed-bigint, boolean, finite-number, exact-sum, percentile, and average-denominator semantics, including decimal exponents beyond PostgreSQL numeric scale. SQL-native step readers use a bounded lateral expansion of only the first 10,000 array entries, matching the fallback normalizer’s first-10,000 slice rather than expanding an unbounded persisted array.

The LLM-usage endpoint returns at most the lexicographically first 64 model buckets. by_model_limit publishes that cap and by_model_truncated reports when additional buckets were omitted. Run-wide token and cost totals remain authoritative across the complete population, while every retained model bucket keeps exact counts and sums.

The execution-latency endpoint likewise returns at most the lexicographically first 64 step-type buckets. by_step_type_limit publishes that cap and by_step_type_truncated reports when additional buckets were omitted. PostgreSQL computes exact continuous percentiles; the non-PostgreSQL development fallback uses an order-independent, fixed-size sample per retained bucket and sets percentiles_approximate whenever any returned percentile is sampled. Counts remain exact for the overall population and each retained step type.

Execution evidence rollups are complete replacements of one closed period. On PostgreSQL, one statement uses a transaction-scoped advisory lock and a single snapshot for projection, aggregation, ranking, upsert, and deletion of stale buckets. Stale deletion is scoped to the same grain and period, including an empty recomputation, so one tick cannot erase another window.

model_config_ref is canonical only when it contains 1–4096 storage-safe Unicode characters. Execution writes and list, detail, cost, and leaderboard reads use that contract; the control-plane and Knowledge Assistant OpenAPI mirrors and portal clients publish the same constraints on the shapes they expose. Rollup identity uses a fixed-width tagged digest rather than indexing the full reference. A missing reference is displayed as "unspecified" with model_config_ref_missing: true; a real literal "unspecified" has model_config_ref_missing: false and a different digest, so the two buckets never collide. If migration 226 is downgraded into the legacy raw-key schema and both buckets exist, it deterministically retains the historical missing bucket and removes the literal bucket that the old key cannot represent separately.

These bounds and scope fields are propagated through the control-plane and Knowledge Assistant OpenAPI documents, generated portal clients, live portal views, and the compose-backed telemetry trust-boundary probe. Orqestra is unreleased, so v2 uses the canonical contract directly: there is no dual-write, legacy-field shim, or v2-to-legacy fallback.

Cost Attribution APIs

Per-Execution Observed Usage

GET /v1/metrics/llm/usage/executions returns bounded per-execution model attempt and retrieval metadata for an explicit workspace and recorded-time window of up to 30 days. It requires both execution.view and cost.view. The window uses each execution’s recorded created_at timestamp, includes the start, and excludes the end; results are ordered by timestamp and execution ID. Provider-reported committed attempts count as confirmed use, including zero-cost facts. Proven no-dispatch attempts are excluded; reservations, estimate-only, and unresolved facts remain uncertain. The response omits cost amounts and retrieval query/source payloads. Knowledge-base references come from recorded retrieval events, and tool usage is marked not covered because no producer currently records it. This read does not change the existing 64-model aggregate at GET /v1/metrics/llm/usage. Pages expose generation time, detail truncation, and the count of unattributed attempt facts in the same workspace and time window; the unattributed count uses each attempt fact’s recorded timestamp. A page returns at most 100 executions and pagination stops after 10,000 executions in a window. next_offset is present only when another page can be fetched; result_cap_reached is true when additional executions exist beyond the 10,000-row horizon. truncated remains true when the response omits executions or bounded evidence details.

Execution Cost

curl http://localhost:8000/v1/executions/{execution_id}/cost

Workload Cost Report

curl http://localhost:8000/v1/workloads/{workload_id}/cost-report?window_days=30

Workspace Cost Report

curl http://localhost:8000/v1/workspaces/{workspace_id}/cost-report?window_days=30

Namespace Cost Report

curl http://localhost:8000/v1/namespaces/{namespace_id}/cost-report?window_days=30

These endpoints are implemented in the control-plane API. Execution, workload, and workspace cost reporting are also surfaced through the Python SDK.

For ledger-level evidence, GET /v1/workspaces/{workspace_id}/cost-attempts returns a versioned page of stored attempt facts and is available through client.namespaces.list_workspace_cost_attempts(...). It requires workspace cost.view, accepts workload and execution filters, plus step_path (a JSON array of at most eight step ids, root first) to list one node of a workload’s step tree and its descendants, and defaults to a 30-day window (maximum 365 days) measured on each attempt’s own creation time, over every purpose: each fact carries a purpose tag (production, evaluation, experiment or pre_release). The replay winner is decided over the whole request, so a fact’s disposition does not depend on the window or the page. Each fact distinguishes its settled contribution, replay-excluded evidence, or outstanding liability; nullable amounts and stored attribution stay nullable. GET /v1/workspaces/{workspace_id}/cost-attempts/export streams the same window as CSV (client.namespaces.export_workspace_cost_attempts_csv(...)); a window holding more than 200,000 facts is refused with 413, so narrow it. Continuations bind the workspace, filters, window, page size, and position. They are live reads rather than a snapshot, so a late commit can change a fact’s disposition between pages.

Each fact also carries release_id, the release of its execution, and null when the attempt names no execution or one outside the workspace. The attribution is computed when you read; it is not stored on the attempt. Two FinOps reads build on it, both requiring cost.view and the FinOps module: GET /v1/finops/workspaces/{workspace_id}/workloads/{workload_id}/releases (client.finops.get_workload_release_history(...)) pages a workload’s releases, newest first, with the cost per execution of their production attempts (measured) beside the cost of their own test and evaluation runs (estimated), the latest terminal evaluation result per suite (its status, and the pass rate unless it errored or is incomplete), and the first production attempt in the window (not an activation time; none is recorded). GET .../releases/diff (client.finops.get_workload_release_diff(...)) lays release B’s step spend against release A’s in one window, with attempts under neither release in other, so the three add up to the window total, and lists which bindings (by role name and count) and model-config keys changed; full binding detail stays behind release.view. A release with no executions of a kind shows no figure and says why. Managed releases are not listed and answer 404 on the diff. The release readiness cost check is advisory: it warns when a candidate’s test and evaluation cost per execution is more than 10% above the test and evaluation cost per execution of the release it replaces over the last 30 whole UTC days, and it says “not evaluated” when either side has no priced test or evaluation execution. It never blocks activation.

For incident cohorts, the same contract is exposed by cost_status, cost_sources, cost_source_subtotals, and cost_coverage. The cohort total_cost_usd is a known canonical subtotal and may be null; the coverage counts explain whether the cohort is complete, partial, or not recorded.

What To Rely On Today

Safe claims for the current platform:

• request/trace correlation is present
• workload, execution, release, tenant, workspace, and provider-route correlation is present when trusted server-side context has resolved those values
• Prometheus metrics are present
• execution-level telemetry summaries are persisted
• execution, workload, workspace, and namespace cost reports are available

Claims that should be treated more cautiously:

• polished operator dashboards for every telemetry dimension
• rich replay and trace visualization beyond the existing evidence/execution APIs
• highly customized per-flow observability semantics beyond the current schema

Best Practices

Turn on tracing intentionally. Use trace_content sparingly when payloads may contain sensitive information.

Use telemetry summaries for cost governance. They are the stable, persisted source for execution-level cost analysis.

Correlate logs, traces, and evidence. Orqestra is strongest when operators use all three together rather than treating observability as only a metrics problem.