ElevenLabs Voice Memory

Three tiers, one package

Requirements

Tier 0 needs only the first two rows — no server, no code. Python 3.10+ for Tiers 1–2.

Tier 2 — the memory proxy

pip install elevenlabs-memorysync

# server.py — deploy anywhere ElevenLabs can reach
from elevenlabs_memorysync import create_proxy_app

app = create_proxy_app(
    api_key="ms_...",                  # MemorySync (or MEMORYSYNC_API_KEY)
    upstream_api_key="sk-...",         # your LLM key (or OPENAI_API_KEY)
    # upstream_base_url="https://api.groq.com/openai/v1",  # any provider
    proxy_api_key="a-long-random-secret",
)
# uvicorn server:app --host 0.0.0.0 --port 8013

# What the proxy's capture performs on the wire — one call per caller
# turn. The server extracts the durable facts in it and stores only
# those, grouped under the elevenlabs:: session scope. The model's
# replies are not sent.
curl --request POST https://api.memorysync.io/v1/memory/add_turn \
  --header "X-API-Key: $MEMORYSYNC_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{"tenant_id":"acme","user_id":"+15551234567","source":"elevenlabs","text":"I always take the window seat","role":"user","speaker":"human@elevenlabs::+15551234567#h<content-hash>","metadata":{"session_id":"elevenlabs::conv_abc"}}'

In the agent: LLM → Custom LLM, Server URL = your deployment, Model ID = the id of your chosen upstream model, API key = your proxy_api_key. Every turn is enriched with recalled memories under the budget, and the caller’s turn is sent for fact extraction with an idempotency seed — a slow or dead memory backend degrades to an unenriched turn, never a delayed reply. The model’s replies are relayed to ElevenLabs but not sent to MemorySync.

Who is calling? The identity ladder

How a turn flows through the proxy

Tier 1 — session-start context, no proxy

from elevenlabs_memorysync import fetch_memory_variables

# When your app starts a session (web SDK / widget / SIP):
variables = await fetch_memory_variables(user_id="customer-7")
# {"memory_summary": "Prefers aisle seats. Vegetarian. ..."}

conversation = Conversation(
    client, AGENT_ID,
    conversation_initiation_client_data={
        "dynamic_variables": variables,
    },
)

Nothing runs in the voice path — memory is fetched once at session start and interpolated into the prompt. Pair it with the post-call webhook below for capture, and the memory is one call stale at worst: perfect when you cannot host a proxy but want recognized callers.

Post-call capture (tiers 1 and 2)

from elevenlabs_memorysync import create_webhook_app

app = create_webhook_app(
    api_key="ms_...",
    webhook_secret="wsec_...",   # or ELEVENLABS_WEBHOOK_SECRET
)
# Configure the URL under Agents → Settings → Post-call webhooks

Signatures are verified with the exact official scheme (t=…,v0=HMAC-SHA256, 30-minute tolerance — pinned against the ElevenLabs SDK in CI) plus a constant-time compare. The caller’s turns from the transcript are sent to fact extraction with the same idempotency seeds the proxy uses, so running both gives live memory plus a sweep that catches anything missed — a turn the proxy already sent is recognised and never extracted twice, by test. The agent’s turns are not sent. The receiver answers with a summary of the call’s transcript entries — sent, stored (the same count), already, skipped, errors — and no memory ids, since facts are extracted asynchronously and get their own. A poison payload gets a 200 skip (the webhook can never be auto-disabled by one bad event); only a fully-failed delivery returns 500 so ElevenLabs redelivers.

Tier 0 — zero code, dashboard only

No server at all — the agent calls MemorySync directly when the model decides to look something up. That trade-off (LLM-gated recall) is exactly why the proxy tier exists; start here, graduate up the ladder.

Configuration (proxy)

Example: what the caller experiences

What happens server-side

Verify it is working

# list what the agent remembers about a caller
curl "https://api.memorysync.io/v1/memory/acme/+15551234567/list?limit=20" \
  --header "X-API-Key: $MEMORYSYNC_API_KEY"

Or open Memory Explorer and filter by the caller id. This whole path is verified against a real ElevenLabs agent in production — agent created via the platform API, Custom LLM pointed at the proxy, two live conversations: facts landed under the prompt-tag identity, the second call answered from memory, and the tag was never spoken.

Troubleshooting

Privacy and data safety

Only extracted facts persist — the caller’s turns are processed and discarded, and the agent’s replies are never sent. The injected memory block is capture-excluded, so recalled context never re-enters storage. Workspace secrets (secret__*) never reach the LLM; the prompt tag is stripped before the upstream call so an identity can never be spoken aloud. Deletion is a first-class API (/memory/forget) and dashboard action, per end user. Audio never reaches MemorySync — text only.

Supported versions

Where to go next