How to make AI agent memory auditable
Make agent memory an append-only log of who wrote what, from which source, and which memories each run read; ask a person before a memory can change what the agent does.
· 8 min read
Agent memory is auditable when you can answer three questions about any memory: who or what wrote it, from which source, and which later runs read it. The way to get there is to stop treating memory as a table the agent edits in place, and to treat it as an append-only log of events, with the searchable store built from that log.
It matters because memory is where an agent’s past turns into its future behaviour. A sentence planted in one email can sit in memory for weeks and steer every run that retrieves it. OWASP lists this as ASI06, Memory and Context Poisoning, in its Top 10 for Agentic Applications, and asks for validated writes, recorded sources, and memory separated per user and task.
Memory as a log of events
Five kinds of event are enough. A memory is written, not stored, retracted, erased or read. Nothing is updated in place: a correction is a new memory that supersedes the old one, so the old one and the reason for the change stay visible.
// Memory is a log of events. The searchable index is built from it.
type Source = { type: "person" | "agent" | "tool" | "web" | "email"; ref: string };
type Written = { kind: "memory.written"; id: string; at: string; text_hmac: string;
source: Source; read?: Source[]; run: string; agent_key_id: string;
supersedes?: string; approval?: string };
type MemoryEvent =
| Written
| (Omit<Written, "kind" | "approval"> & { kind: "memory.not_stored"; approval: string; status: string })
| { kind: "memory.retracted"; id: string; at: string; by_user_id: string; reason: string }
| { kind: "memory.erased"; id: string; at: string; by_user_id: string; request: string }
| { kind: "memory.read"; at: string; run: string; ids: string[] };- Source
- Where the text came from: the person in the conversation, a tool result, a web page, an email, another agent. With a reference, such as the message id.
- Run and key
- Which run wrote it, and with which credential, so a write can be traced back to the conversation and the agent that made it.
- Hash
- A keyed hash (HMAC) of the text, with a key per person. The log can prove what a memory said while the key exists, and nobody can confirm a guess once it is deleted.
- Approval
- For a write that needed a person's yes, the id of that approval.
Ask before a memory can steer the agent
Most memory writes are harmless, and asking about each one would bury people in questions. The dangerous ones share a pattern: the text did not come from the person the agent works for, and it will be shown to future runs. “The customer prefers email” read from a CRM is fine. “Invoices for Acme now go to this new account” read from an email is how an invoice ends up paid to the wrong account.
So the code below has two doors. One is called by your app when the person asks it to remember something, such as a Remember button, and the model cannot reach it. The other is the model’s own tool, and it always asks a person first. The approver sees what the run had read, as your runtime recorded it, never a source the model names: an injected email could simply claim to come from the CRM, or from the person.
import { createHmac, randomUUID } from "node:crypto";
import { ApprovalClient } from "@variantgate/agent";
import { ChangedApprovalError, NotApprovedError } from "@variantgate/agent/errors";
const approvals = new ApprovalClient({
agent: "assistant",
apiUrl: "https://api.variantgate.com",
key: process.env.VG_KEY!,
});
// Keyed per person, so a short fact cannot be confirmed by guessing once its key is gone.
const hmac = (subject: string, text: string) =>
createHmac("sha256", keyFor(subject)).update(text).digest("hex");
// Called by your app when the person asks it to remember something, such as a
// "Remember this" button. The model cannot reach this function.
export async function rememberForPerson(run: Run, text: string, supersedes?: string) {
const old = await replaceable(run, supersedes);
if (old === null) return "unknown memory";
const event = { ...entry(run, text, { type: "person", ref: run.userId }), supersedes };
await store(run, event, text);
await index(run, event, text);
return "stored";
}
// The model's tool. It always asks. What the run had read comes from your runtime's
// own record (run.inputs), never from an argument the model passes in.
export async function remember(run: Run, text: string, supersedes?: string) {
const old = await replaceable(run, supersedes);
if (old === null) return "unknown memory";
const event = { ...entry(run, text, { type: "agent", ref: run.id }), supersedes,
read: run.inputs }; // such as [{ type: "email", ref: "msg_8812" }]
let stored: Written | undefined;
try {
await approvals.guard(
{ action: "memory.write",
summary: `Remember for future runs: "${text.slice(0, 200)}"`,
details: { text, replaces: old ?? "nothing", run: run.id,
read_in_this_run: run.inputs.map((input) => `${input.type}: ${input.ref}`) } },
async (decision) => {
const written = { ...event, approval: decision.id };
await store(run, written, text);
stored = written; // the log is the source of truth from here
},
{ timeoutMs: 15 * 60_000 },
);
} catch (error) {
// Once stored, only reporting the outcome failed: go on, never write it twice.
if (!stored && !(error instanceof NotApprovedError)) {
await memoryLog.append({ ...event, kind: "memory.not_stored", approval: "unknown", status: "error" });
throw error;
}
if (!stored && error instanceof NotApprovedError) {
// Denied, expired, withdrawn, paused, or approved only with changes: not stored.
const status = error instanceof ChangedApprovalError ? "changed" : error.decision.status;
await memoryLog.append({ ...event, kind: "memory.not_stored", approval: error.decision.id, status });
return "not stored";
}
}
// Outside the approval, so an index failure is never reported as a failed write.
// A rebuild of the index from the log picks up anything it missed.
if (stored) await index(run, stored, text).catch(() => {});
return "stored";
}
// The text a correction replaces, so the approver sees it. Undefined when nothing is
// replaced, null when the id is not one of this person's memories for this task.
async function replaceable(run: Run, id?: string) {
if (!id) return undefined;
const hit = await memoryIndex.get(id);
if (!hit || hit.subject !== run.userId || hit.task !== run.task) return null;
return (await memoryText.get(id)) ?? null;
}
function entry(run: Run, text: string, source: Source) {
return { kind: "memory.written" as const, id: randomUUID(), at: new Date().toISOString(),
text_hmac: hmac(run.userId, text), source, run: run.id, agent_key_id: run.keyId };
}
async function store(run: Run, event: Written, text: string) {
await memoryText.put(event.id, text, { subject: run.userId }); // erasable, see below
await memoryLog.append(event);
}
async function index(run: Run, event: Written, text: string) {
await memoryIndex.add(event.id, text, // the index keeps an embedding, not the text
{ subject: run.userId, task: run.task, source: event.source, at: event.at });
if (event.supersedes) await memoryIndex.remove(event.supersedes); // still in the log
}- The person sees the full text and its source. In VariantGate, characters a reader cannot see, such as Unicode tag characters used to hide instructions, are shown marked with a warning rather than hidden.
- A no is recorded too, with its status. A memory a person denied is evidence that something tried to plant it, worth a look at its source. One nobody answered in time is just not stored.
- Waiting can happen off the run. If fifteen minutes is too long to hold a run open, queue the write and let the run go on without it. Store it when the answer arrives.
If too many writes ask, narrow the rule rather than dropping it: only memories that name accounts, addresses, recipients, amounts or standing instructions, or only sources from outside your organisation. A standing allow rule for memory.write per agent key can answer the rest, recorded in the name of the owner who wrote it.
Record which memories each run read
Writes tell you what memory holds. Reads tell you why an agent did something. Log the ids of the memories retrieved for each run, and show them to the model as quoted notes with their source, not as instructions:
export async function recall(run: Run, query: string) {
// Only this person's memories for this kind of task.
const hits = await memoryIndex.search(query, { subject: run.userId, task: run.task, limit: 8 });
const notes = [];
for (const hit of hits) {
const text = await memoryText.get(hit.id); // gone once erased
if (text !== undefined) notes.push({ id: hit.id, note: text, source: hit.source, written: hit.at });
}
// Which memories were in front of the model, for this run.
await memoryLog.append({ kind: "memory.read", at: new Date().toISOString(), run: run.id,
ids: notes.map((n) => n.id) });
// Shown to the model as quoted notes with their source, never as instructions.
return notes;
}
export async function retract(id: string, byUserId: string, reason: string) {
await memoryIndex.remove(id);
await memoryLog.append({ kind: "memory.retracted", id, at: new Date().toISOString(),
by_user_id: byUserId, reason });
}
// An erasure request covers every memory about the person, corrected ones included.
export async function eraseSubject(userId: string, byUserId: string, request: string) {
for (const id of await memoryText.idsFor(userId)) {
await memoryIndex.remove(id);
await memoryText.delete(id); // the log keeps the keyed hash, the source and the reads
await memoryLog.append({ kind: "memory.erased", id, at: new Date().toISOString(),
by_user_id: byUserId, request });
}
await deleteKeyFor(userId); // the hashes can no longer be checked against a guess
}Now the question “why did the agent send the invoice to that account?” has a path to its answer:
- Find the run in the tool-call log by the payment it made.
- Look up the run’s
memory.readevent for the ids it retrieved. - Read each memory’s
memory.writtenevent: its source, the run that wrote it, and the approval, if any. - Retract the bad memory with a reason, which also takes it out of the index, and find every other run that read it.
Erasing without losing the trail
Memory often holds personal data, and people may ask for it to be erased. An append-only log and erasure fit together if the text lives apart from the events. Keep each memory’s text in a store keyed by the person it concerns, or encrypted with a key per person, and keep only ids and hashes in the log. Erasing deletes the text, or the key, and the memory’s index entry, and recall skips any memory whose text is gone. The log still shows that a memory existed, where it came from and which runs read it, without saying what it was. A plain hash of a short sentence could be confirmed by guessing, which is why the log keeps a keyed hash and erasure deletes the person’s key.
One exception: the text sent for approval stays in the approval record, because a sealed record cannot lose an entry without breaking every seal after it. If memories may hold personal data, keep it out of the ones that ask, or send the person a reference they can open in your own system instead of the text.
A short checklist
- Every memory has a source, a run and a credential.
- Nothing is edited in place; corrections supersede.
- Text from outside the conversation does not become memory without a check, and the high-impact kinds need a person.
- Every run records which memories it read.
- Memory is separated per user and per task.
- Text can be erased while the events stay.
Memory is one part of what an agent leaves behind. The rest, and how the parts join up, is in what an audit trail for AI agents should record. The other agentic risks, and where an approval helps with each, are in our overview of the OWASP Top 10 for Agentic Applications.