ElevenLabs Voice Memory
Long-term memory for ElevenLabs Agents — returning callers are recognized on the web and on the phone. Three integration tiers in one package: a zero-code memory tool, session-start context with post-call capture, and a memory-injecting Custom LLM proxy with a hard latency budget.
Three tiers, one package
| Tier | What runs | Memory freshness | Code needed |
|---|---|---|---|
| 0 — Zero code | An ElevenLabs webhook tool calls the MemorySync REST API directly | When the LLM decides to look | None — dashboard only |
| 1 — No proxy | fetch_memory_variables at session start + the post-call webhook receiver | Start of call | ~5 lines |
| 2 — Full proxy | create_proxy_app as the agent’s Custom LLM | Every turn, under a hard budget | ~3 lines |
| Mem0 | Supermemory | Zep | MemorySync | |
|---|---|---|---|---|
| ElevenLabs integration | △ client-tools recipe — recall only if the LLM decides to call | ✗ none at all | △ proxy example, copy-paste | ✓ elevenlabs-memorysync package, three tiers |
| Recall latency budget | ✗ unbounded await | — | ✗ unbounded await | ✓ hard 1.2s default, tested against a 5s-slow backend |
| Phone-caller identity | ✗ static env var (single user!) | — | ✗ browser localStorage only | ✓ prompt-tag reads {{system__caller_id}} — zero client code |
| Upstream LLM choice | n/a | — | ✗ OpenAI hard-coded | ✓ any OpenAI-compatible provider |
| Post-call webhook capture | ✗ unused | — | ✗ unused | ✓ HMAC-verified, converges with live capture at zero duplicates |
| Signature verification | n/a | — | shared secret only | ✓ pinned byte-for-byte to the official SDK scheme, constant-time compare |
Tier 2 — the memory proxy
pip install elevenlabs-memorysync
In the agent: LLM → Custom LLM, Server URL = your deployment, Model ID = the upstream model (e.g. gpt-4o-mini), API key = your proxy_api_key. Every turn is enriched with recalled memories under the budget and both sides of the conversation persist with idempotency seeds — a slow or dead memory backend degrades to an unenriched turn, never a delayed reply.
Who is calling? The identity ladder
| Rung | Mechanism | Works for |
|---|---|---|
| 1 | customLlmExtraBody: { user_id } at session start (enable *Custom LLM extra body* in the agent’s Security tab) | Web apps and SDKs where your code knows the user |
| 2 | One line in the agent’s system prompt: memorysync-user: {{system__caller_id}} — the proxy extracts the interpolated value and strips the line before the model sees it | Phone calls (Twilio/SIP), the widget, everything — zero client code |
| 3 | default_user_id= fallback | Single-user kiosks and demos |
Post-call capture (tiers 1 and 2)
from elevenlabs_memorysync import create_webhook_appapp = create_webhook_app(api_key="ms_...",webhook_secret="wsec_...", # or ELEVENLABS_WEBHOOK_SECRET)# Configure the URL under Agents → Settings → Post-call webhooks
Signatures are verified with the exact official scheme (t=…,v0=HMAC-SHA256, 30-minute tolerance — pinned against the ElevenLabs SDK in CI) plus a constant-time compare. The whole transcript stores with the same idempotency seeds the proxy uses, so running both gives live memory plus a sweep that catches anything missed — at zero duplicates, by test. A poison payload gets a 200 skip (the webhook can never be auto-disabled by one bad event); only a fully-failed delivery returns 500 so ElevenLabs redelivers.
Tier 0 — zero code, dashboard only
| Field | Value |
|---|---|
| Tool type / name | Webhook · search_memory |
| Method + URL | POST https://api.memorysync.io/memory/query |
Header X-API-Key | {{secret__memorysync_api_key}} (workspace secret — never reaches the LLM) |
Header X-End-User-ID | {{user_id}} (dynamic variable) |
| Body schema | query (string — what to look up), k (integer, default 5) |
No server at all — the agent calls MemorySync directly when the model decides to look something up. That trade-off (LLM-gated recall) is exactly why the proxy tier exists; start here, graduate up the ladder.
Configuration (proxy)
| Parameter | Default | Meaning |
|---|---|---|
upstream_base_url | https://api.openai.com/v1 | Any OpenAI-compatible provider |
proxy_api_key | none | Bearer secret ElevenLabs must present — set it in production |
recall_timeout | 1.2 | Hard recall budget in seconds |
top_k | 5 | Memories injected per turn |
min_prompt_chars | 8 | Skip recall for trivial utterances |
buffer_words | none | e.g. "One moment… " — spoken filler emitted only when the turn is already slow |
buffer_after_ms | 900 | How slow is “slow” before buffer words are used |
default_user_id | none | Identity fallback for single-user deployments |
Supported versions
| Surface | Requires | Verified on |
|---|---|---|
elevenlabs-memorysync 1.0.0 | Python 3.10+, FastAPI; an ElevenLabs agent with Custom LLM and/or post-call webhooks | 36 CI checks driven over the real ASGI apps: SSE relay with tool-call deltas split at awkward byte boundaries, hard budget vs a 5s-slow backend, prompt-tag extraction/stripping, both quota modes, webhook HMAC cross-checked against the official elevenlabs SDK at latest, and the zero-duplicate sweep-convergence proof |