Automatic Secret Redaction

How Engram prevents API keys, credentials, and PII from leaking into long-term agent memory

Published May 2026

The problem: agents see everything

AI coding agents work inside your development environment. They read .env files, parse config blocks, handle API responses with embedded tokens, and discuss database connection strings in natural conversation. Every one of these secrets passes through the conversation as plaintext.

When you add persistent memory to an agent, you create a new attack surface: the memory store. A conversation that mentions sk-ant-api03-... in passing becomes a searchable, retrievable record. If the memory store is compromised, every secret that ever appeared in a conversation is exposed — not because someone targeted your secrets, but because the memory system stored everything indiscriminately.

This is not a hypothetical. Consider what a typical Claude Code session contains:

[user]: Can you check why the deploy is failing?
[assistant]: Let me read the .env file...
  STRIPE_SECRET_KEY=sk_live_51Abc...
  DATABASE_URL=postgres://admin:p4ssw0rd@db.example.com:5432/prod
  GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxx

Without redaction, that entire block — including three production secrets — would be stored verbatim in the memory system, embedded as a vector, and returned in future search results. The secret that was relevant for 30 seconds during debugging becomes a permanent, searchable record.

Engram’s approach: redact before storage

Engram runs an automatic redaction pipeline on every message before it touches persistent storage. The pipeline strips secrets, credentials, and PII from message content, replacing them with [REDACTED] markers. This happens at the application layer, before compression, before chunking, before embedding, and before writing to D1.

Message arrives via append_messages
  |
  v
[1] Redaction pipeline        <-- secrets removed here
  |
  v
[2] Gzip compression          <-- compressed text is already clean
  |
  v
[3] Write to D1 (messages)    <-- no secrets in the database
  |
  v
[4] Chunk into windows        <-- chunks contain [REDACTED] markers
  |
  v
[5] Generate embeddings       <-- vectors encode meaning, not secrets
  |
  v
[6] Store in Vectorize        <-- search index is clean

The critical insight: redaction happens once, at ingestion time, and everything downstream inherits the protection. Chunks, embeddings, search results, and API responses all return redacted content. There is no code path where a raw secret can be retrieved from storage after redaction.

What gets redacted

The redaction engine detects seven categories of sensitive data, applied in a specific order to handle overlapping patterns correctly.

1. PEM private keys

Multi-line PEM blocks (RSA, EC, DSA, OpenSSH) are detected and removed in full.

-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA0Z3VS5JJcds3xfn/ygWp...
-----END RSA PRIVATE KEY-----

2. Provider API keys

Pattern-matched by their distinctive prefixes. Currently detected:

3. JWTs

JSON Web Tokens are detected by their three-segment base64url structure (eyJ...).

4. Connection strings

Database and message broker URIs with embedded credentials:

postgres://admin:password@db.example.com:5432/mydb
mongodb+srv://user:pass@cluster.mongodb.net/db
redis://default:secret@cache.example.com:6379
amqps://user:pass@rabbitmq.example.com

5. Secret assignments

Key-value pairs where the key name suggests a secret:

DATABASE_PASSWORD=hunter2
STRIPE_SECRET_KEY="sk_live_..."
api_token: "abc123..."
jwt_secret = "my-signing-key"

The pattern matches common naming conventions: password, passwd, secret, token, api_key, apikey, private_key, access_key, auth_token, client_secret, signing_key, encryption_key, jwt_secret.

6. PII patterns

7. Generic high-entropy tokens

Catch-all patterns for secrets that don’t match a known provider:

• Hex tokens: 32+ character hexadecimal strings
• Base64 tokens: 40+ character base64 strings

These are applied last to catch tokens that slipped through the provider-specific patterns without over-matching on normal text.

Pattern ordering matters

The redaction engine applies patterns in a specific order — most specific first, most generic last. This prevents false positives and ensures that a Stripe key like sk_live_51abc123 is matched by the Stripe-specific pattern (which knows the exact format) rather than the generic base64 catch-all (which would match but with less precision).

1. PEM private keys (multi-line, most distinctive)
2. Provider API keys (prefix-specific, high confidence)
3. JWTs (three-segment structure)
4. Connection strings (URI scheme)
5. Secret assignments (key name heuristic)
6. PII (SSN, credit card, email, phone)
7. Generic high-entropy tokens (catch-all, lowest confidence)

Each step operates on the output of the previous step. If a Stripe key was already replaced with [REDACTED] in step 2, the generic hex matcher in step 7 won’t see it. This layered approach minimizes false positives while maximizing coverage.

What the embedding model sees

After redaction, the text that gets embedded looks like this:

[user]: Can you check why the deploy is failing?
[assistant]: Let me read the .env file...
  STRIPE_SECRET_KEY=[REDACTED]
  DATABASE_URL=[REDACTED]
  GITHUB_TOKEN=[REDACTED]

The embedding model (bge-base-en-v1.5) generates a vector that captures the semantic meaning — “debugging a deploy failure, checking environment variables” — without encoding the actual secret values. When a future search matches this chunk, the agent gets the context (what was being debugged and why) without the credentials.

This is a meaningful distinction: the memory is useful for recall (“we debugged a deploy failure related to env vars”) but useless for credential theft (“what’s the Stripe key?”). The semantic value is preserved while the literal value is destroyed.

Design decisions

Why redact instead of encrypt?

Application-level encryption (ALE) is the obvious alternative: encrypt message content with a per-org key, decrypt on read. Engram’s issue #70 explores this approach in detail. ALE has real value for defense-in-depth, but it doesn’t solve the same problem as redaction:

Redaction is the right tool for preventing secrets from persisting in memory. Encryption is the right tool for protecting the entire data store from unauthorized access. They’re complementary, not competing.

Why destroy instead of mask?

Some systems mask secrets reversibly — storing the original but displaying sk_live_****. Engram destroys the secret: the original value is never written to D1, never embedded, never stored anywhere. This is a deliberate choice:

1. No key management. Reversible masking requires storing the original somewhere, which creates a new secret to protect.
2. No accidental exposure. A bug in the display layer can’t leak what doesn’t exist.
3. Simpler compliance. For GDPR and SOC 2, it’s easier to prove “we don’t store secrets” than “we store secrets but they’re masked.”

The trade-off: if an agent needs the actual secret value later, it has to read the source (the .env file, the config service) again. This is the right trade-off for a memory system — memory should store what happened, not the credentials used while it happened.

Why redact at ingest, not at query time?

Query-time redaction would let you store the raw text and strip secrets only when returning results. This is worse in every way:

1. Embeddings would encode secrets. The vector for “STRIPE_SECRET_KEY=sk_live_51abc” would be numerically close to the vector for “sk_live_51abc” alone. A targeted search could surface the secret.
2. FTS5 would index secrets. A keyword search for sk_live would return the raw text.
3. Database contains plaintext secrets. Any access to D1 (admin, backup, migration) exposes them.
4. Performance. Redacting on every read is slower than redacting once on write.

Ingest-time redaction means the secret never enters the system. Every downstream component — storage, indexing, embedding, search, API response — inherits the protection automatically.

Limitations and future work

False negatives

The redaction engine uses pattern matching, not semantic understanding. It can miss:

• Custom-format tokens that don’t match known provider prefixes or generic entropy patterns
• Secrets embedded in URLs that aren’t connection strings (e.g., https://api.example.com?token=abc123)
• Passwords in natural language (“the password is hunter2” — the assignment pattern catches password=hunter2 but not conversational mentions)

False positives

The generic high-entropy patterns (hex 32+, base64 40+) can match non-secret values:

• Git commit hashes (40-character hex)
• UUIDs (32 hex characters with dashes removed)
• Base64-encoded content that isn’t a secret (e.g., image data URIs)

The layered ordering minimizes this: specific patterns run first, and the generic catch-all only fires on strings that didn’t match anything else.

Planned improvements

• Application-level encryption (issue #70) for defense-in-depth on the storage layer
• Configurable redaction — let organizations add custom patterns for internal token formats
• Redaction audit log — record what was redacted (pattern type, count) without recording the value
• Client-side redaction — optional SDK that redacts before the message leaves the agent, for zero-trust ingest

Summary

Engram’s automatic secret redaction is a pre-storage pipeline that strips API keys, credentials, connection strings, PII, and high-entropy tokens from every message before it’s written to the database, chunked, or embedded. It runs seven detection layers in order from most specific to most generic, replacing matches with [REDACTED] markers. The semantic meaning of conversations is preserved for search while the literal secret values are destroyed at ingest time.

For agents that work with production infrastructure — reading configs, debugging deploys, handling API responses — this is the difference between a memory system that’s a security liability and one that’s safe by default.