Accepted · ICML 2026 Workshop on SCALE
Before It Persists: Write-Time Defense for Multimodal Agent Memory
SAGE-Mem moves persistent-memory security to the write boundary, separating transient evidence from durable belief before poisoned context can compound.
Abstract / overview
Persistent memory gives multimodal agents continuity across conversations, browser sessions, documents, images, and tools. It also creates a compounding attack surface: once unsupported content becomes durable state, later retrieval and consolidation can reuse it as if it were reliable evidence.
SAGE-Mem is a governed layer between heterogeneous observations and a downstream planner. It controls what may be written, what earns promotion from evidence into durable belief, and what retrieved memory may support future reasoning. Led by Agam Pandey, Scriben's Founding AI Researcher, the work was accepted to the ICML 2026 Workshop on SCALE.
Research problem
Retrieval-time filtering acts after an observation has already entered memory. By then, poisoned content may have been summarized, reinforced, merged with clean evidence, or retrieved repeatedly until its origin becomes difficult to inspect.
Multimodal systems make the boundary harder: OCR and a caption derived from the same image can look like two agreeing sources even though they are dependent views of one observation. A robust memory system must reason about support, independence, conflict, and provenance before persistence.
Approach
A write-time admission guard classifies incoming observations as data, directives, or metadata and blocks unsupported or injected content before storage. Admitted observations remain typed evidence rather than becoming belief automatically.
Evidence is promoted into durable belief only when support is sufficient, independent, and non-conflicting. Bayesian channel trust, session-relative anomaly detection, consistency-graph checks, and dependent-evidence handling provide separate signals rather than one opaque confidence score.
At retrieval, a provenance-aware policy applies a channel-trust floor, partition multiplier, and conflict gate before memory can support downstream reasoning. LoCoMo-Adv and MM-BrowseComp-Adv stress the full lifecycle with long-horizon and multimodal browsing attacks.
Research → Scriben system
Where this research becomes Scriben
Retrieval-only filtering acts after unreliable context has already persisted. Scriben applies the research earlier: real-world context is treated as evidence at capture, earns promotion before becoming durable memory, and remains inspectable when recalled.
- 01
Evidence before belief
Transcripts, notes, and extracted commitments enter as source-linked evidence, not as unquestioned truth.
- 02
Govern the write boundary
Context becomes durable only after support, consistency, and conflict checks establish that it should persist.
- 03
Retrieve with provenance
Recalled context keeps a path back to its source, while weak or conflicting evidence can be withheld from downstream reasoning.
- 04
Keep action human-controlled
Memory can prepare answers and work, but consequential writes remain reviewable before they reach another person or system.
The result: Scriben is designed not merely to remember more, but to remember with an evidence boundary, so ambient context can become useful intelligence without becoming unexamined truth.
Results or findings
On LoCoMo-Adv, SAGE-Mem reduced attack write admission from 1.000 for the retrieval-time baseline to 0.004. Retrieval contamination fell from 0.158 to a 95% rule-of-three upper bound of ≤0.002 across the evaluation budget; this is not an exact-zero claim.
The safety gain has a visible utility cost. Benign Completion Under Attack was 0.655 for the retrieval-time baseline and 0.418 for SAGE-Mem. The result supports write-time control as a security boundary, not the claim that its operating point is universally optimal.
On the broader five-attack MM-BrowseComp-Adv suite, BrowseGuard-Extended reduced Write ASR from 0.255 to 0.037 and Retrieval ASR from 0.564 to 0.369 relative to the base SAGE-Mem configuration. The narrower answer-overwrite specialization reached ≤0.003 Write ASR and ≤0.002 Retrieval ASR, but should not be generalized to the full attack suite.
Poster
ICML 2026 SCALE poster. The expanded project page reports additional five-attack results from the current frozen analysis artifacts; the poster preserves the original snapshot.
Open full-resolution poster ↗System diagram
Browser, OCR, vision, documents, chat, and tools
Classify content and quarantine unsupported writes
Require independent, non-conflicting support for belief
Gate reasoning by provenance, trust, and conflict
Limitations
Results are three-seed means on adversarial extensions of LoCoMo and MM-BrowseComp. They establish behavior on these frozen pipelines, not safety across arbitrary models, modalities, deployments, or adaptive attackers.
Reported Retrieval ASR reflects the combined admission, promotion, and provenance-aware retrieval stack. The evaluation does not isolate every layer's causal contribution, and the conservative write boundary can reject useful evidence.
Upper bounds marked ≤ use the rule of three when no events were observed at the available evaluation budget. They should not be described as proof of zero contamination.
This research informs Scriben's evidence-first approach to durable personal intelligence. The research artifact is not presented as a verbatim description of Scriben's production system.
Safety, privacy and deployment considerations
Write-time rejection can be irreversible. A production system should preserve an inspectable quarantine path for uncertain evidence and expose why an observation was blocked, retained, or promoted.
Source trust and anomaly signals can encode institutional bias. Calibration, provenance displays, red-team coverage, and human review remain necessary wherever persistent memory can influence consequential actions.
Citation and links
Suggested citation
Pandey, A. (2026). Before It Persists: Write-Time Defense for Multimodal Agent Memory. ICML 2026 Workshop on SCALE.