Memory & Context Evaluation

Technical reference. This page is for integrations, advanced configuration, or platform operations. For portal tasks, start with the user guide.

The memory and context regression suite continuously checks that governed memory and context assembly keep their safety guarantees: secrets and credentials are never recalled, stale or cross-scope memory is suppressed, disabled/forgotten memory stays out of recall, injected retrieval is quarantined, and critical context survives budget pressure.

It is a deterministic, test-side suite — it runs the real recall-policy and budget-planner services against fixed fixtures, with no database, model calls, or randomness, so a regression is reproducible and actionable by scenario name.

The full cross-surface contract is in docs/reference/memory-context-evaluation-harness.md.

Running the suite


make eval-memory-context

This runs the 22 scenarios, writes one verifiable-reward row per scenario into eval-output/, and asserts the five suite metrics meet their configured floors. The command exits non-zero if any scenario fails or any metric drops below its floor.

Reading the output

The summary lists each metric, its observed value, and its floor:


memory-context eval: 22 scenarios, 0 failed, 0 skipped
  recall_precision: 1.000 (floor 1.0) ok
  unsafe_recall_rate: 1.000 (floor 1.0) ok
  stale_suppression_rate: 1.000 (floor 1.0) ok
  context_completeness: 1.000 (floor 1.0) ok
  provenance_availability: unverifiable (floor 0.0) — not gated

A failing scenario prints its id and a bounded diff of expected vs. observed recall/exclusion so the regression can be traced to one invariant.

Scenario categories

Category	Example check
Positive recall	user-confirmed preference is recalled under a permissive policy
Unsafe recall	a `secret`-sensitivity row is excluded from recall
Stale suppression	a `superseded` row is excluded from recall
Purge / disable / forget	a disabled row is excluded after a forget request
Cross-scope isolation	one actor’s memory does not appear in another’s recall
Injection quarantine	a denied retrieval chunk is dropped from context
Budget drop diagnostic	`tool_result` is dropped first; drops appear in diagnostics

Continuous coverage

The same 22 scenarios run as governance pen-test probes under the nightly and audit profiles, so the regression guard is exercised both in local verification and in the scheduled pen-test reporter. Each scenario is a non-negotiable governance regression guard.