Pen-Test Runbook

Technical reference. This page is for integrations, advanced configuration, or platform operations. For portal tasks, start with the user guide.

The Orqestra pen-test suite (packages/orqestra-pentest) runs a 172-probe corpus in four profiles — smoke, nightly, audit, and redteam. This runbook covers how operators run it, how the gates work, and how to respond when a gate fires.

Quick reference

# Run the blocking smoke gate locally (same job CI runs on every PR)
orqestractl pentest run
 
# Regenerate the nightly report against your local compose stack
make dev && orqestractl pentest run --profile nightly
 
# Enforce the gate against the most recent report
orqestractl pentest gate --profile nightly
 
# Validate baseline + suppressions files before committing
orqestractl pentest check

Profiles at a glance

When the smoke gate fails on a PR

1. Read the failing probe id from the CI log (e.g. ORQ-A-HEADERS-001).
2. Find the probe file via rg "ORQ-A-HEADERS-001" packages/orqestra-pentest.
3. Decide:
   • Fix the code — the probe is catching a real regression. Fix it, push, smoke re-runs green. This is the preferred outcome.
   • Baseline the finding — the gap is known, tracked, and acceptable for now. Add an entry to packages/orqestra-pentest/pentest-baseline.json with probe_id, fingerprint (copy from the failing report), issue_url (GitHub issue or follow-up PR), owner, max_age_days (≤ 180 for high, ≤ 365 for medium), and reviewer_approval: true. Commit the baseline in the same PR.
   • Suppress the probe — strongly discouraged. Only for probes that are no longer applicable (e.g. the feature was removed). Suppressions are capped at one extension per entry.
4. Re-run locally: orqestractl pentest run && orqestractl pentest gate.

Baseline hygiene

Baseline entries are regression guards with an explicit expiry. Quarterly (first Monday of the quarter), platform-security runs orqestractl pentest check and reviews any baseline entry where today - first_seen > max_age_days. Each stale entry must be either fixed or re-blessed with a new max_age_days and a comment on the tracking issue explaining why the gap persists.

Full playbook at docs/reference/pen-test-maintenance.md .

Nightly workflow artefacts

.github/workflows/pentest-nightly.yml produces:

• reports/pentest/nightly.json — machine-readable corpus
• reports/pentest/nightly.md — human-readable report
• reports/pentest/nightly.sarif — aggregate SARIF (all findings)
• reports/pentest/nightly-{api,governance,llm,supply_chain,rt}.sarif — per-scanner SARIF, each uploaded with its own GHAS category so retiring one scanner does not orphan alerts from another
• reports/pentest/hash-manifest.txt — SHA-256 of every artefact
• reports/pentest/hash-manifest.sha256 — SHA-256 of the manifest itself

Artefacts retain for 90 days. Findings flow into GitHub Security (“Code scanning alerts”) so oncall can triage without digging into the workflow artifact bundle.

Audit workflow evidence chain

.github/workflows/pentest-audit.yml is operator-triggered only and requires two inputs: reason (why this audit run is being performed, e.g. “Q2 SOC2 control 7.1 evidence”) and reviewer (GitHub handle or email of the reviewer on record). Both are signed into the manifest.

The signing chain:

1. Pytest runs the audit profile, producing JSON/MD/SARIFs.
2. The workflow computes audit-manifest.txt — a plain-text file containing the reason, reviewer, commit, run id, timestamp, and the SHA-256 of every artefact.
3. sha256sum over that file lands as audit-manifest.sha256.
4. cosign sign-blob --yes --output-signature audit-manifest.sig --output-certificate audit-manifest.pem audit-manifest.txt signs keyless via Sigstore’s Fulcio (the workflow’s id-token: write permission). No long-lived keys.
5. Manifest + signature + certificate land in the artefact bundle (365-day retention).

To verify an audit artefact bundle later:

cosign verify-blob \
  --certificate audit-manifest.pem \
  --certificate-identity-regexp 'https://github.com/.+/\.github/workflows/pentest-audit\.yml@.+' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  --signature audit-manifest.sig \
  audit-manifest.txt

Red-team profile

Red-team probes are adversarial: they stop/start services mid-run (fault injection), send injection payloads through RAG and tool-use paths (propagation), and compare observable behaviour across realms (differential drift). They are never run in CI — they are non-deterministic, compose-required, and sometimes destructive to compose state.

Run locally only:

make dev
orqestractl pentest run --profile redteam
 
# Optional: enable garak LLM red-team battery
cd packages/orqestra-pentest && uv sync --extra redteam
orqestractl pentest run --profile redteam -k garak

Red-team results are NOT gated. The operator reads the report and decides whether a finding justifies a follow-up probe in one of the deterministic profiles.