Pipecat Voice Memory
Long-term memory for Pipecat voice pipelines — one FrameProcessor between your context aggregator and LLM enriches every context frame under a hard latency budget and sends only new user turns for fact extraction. Built on the current frame API and tested through Pipecat’s own harness.
What the package provides
| Piece | What it does |
|---|---|
MemorySyncMemoryService | A FrameProcessor you place before your LLM service. Each LLMContextFrame is enriched with relevant memories (as a system message) and its new user turns are sent to fact extraction — then pushed on, enriched or not, on time. |
| Budgeted enrichment | Recall runs under a hard timeout (default 1.2s). A slow or dead memory backend means an unenriched frame, never a stalled voice reply. |
| Delta-only capture | Only user messages not seen before are sent, tracked by deterministic idempotency seeds — O(new), not O(entire context) per frame. Assistant replies are not sent. |
| Lifecycle safety | On EndFrame, queued sends get a bounded window to land so the caller’s last words are never lost; CancelFrame pushes first, then briefly salvages. |
| Mem0 | Supermemory | Zep | MemorySync | |
|---|---|---|---|---|
| Pipecat integration | ✓ in-tree service | △ exists but imports a removed frame API — cannot load on Pipecat 1.x | ✗ none | ✓ pipecat-memorysync package |
| Recall latency budget | ✗ unbounded await before every LLM push | — | — | ✓ hard timeout, default 1.2s, tested against a 5s-slow backend |
| Write amplification | ✗ re-stores the ENTIRE context every frame — O(n²) writes per call | — | — | ✓ delta-only with idempotency seeds — only new user messages are sent, proven by test |
| Injected context re-stored | unguarded | — | — | ✓ the injected block is capture-excluded, tested |
| End-of-call data loss | pending stores can be torn down | — | — | ✓ graceful EndFrame flush, tested |
| Tested via the official harness | △ partial | ✗ | — | ✓ every test runs through pipecat.tests.utils.run_test |
Requirements
| You need | Where to get it | Used for |
|---|---|---|
| MemorySync API key | Dashboard → API Keys (read + write scopes) | Recall and capture |
| Pipecat 1.0+ | pip install "pipecat-ai[webrtc]" (plus the service extras you use, e.g. deepgram, openai, cartesia) | The pipeline runtime |
| STT / LLM / TTS keys | Your chosen providers — the processor is provider-agnostic | YOUR pipeline’s voice loop. Not used by MemorySync. |
| Python 3.10+ | python.org | Runtime |
# your pipeline's own services (LLM/STT/TTS) - never sent to MemorySyncOPENAI_API_KEY=sk-...DEEPGRAM_API_KEY=...CARTESIA_API_KEY=...# the only key MemorySync needsMEMORYSYNC_API_KEY=ms_...
No MemorySync-side setup beyond the key — tenants, embeddings and fact extraction are provisioned automatically on first write.
Install and place it
pip install pipecat-memorysync
from pipecat.pipeline.pipeline import Pipelinefrom pipecat_memorysync import MemorySyncMemoryServicememory = MemorySyncMemoryService(api_key="ms_...", # or MEMORYSYNC_API_KEY env varuser_id="caller-42", # stable end-user idsession_id="call-123", # optional: scope to this call)pipeline = Pipeline([transport.input(),stt,context_aggregator.user(),memory, # ← between aggregator and LLMllm,tts,transport.output(),context_aggregator.assistant(),])
# What capture performs on the wire — one call per NEW user message only.# The server extracts the durable facts in it and stores only those,# grouped under the pipecat:: session scope. Assistant replies are not sent.curl --request POST https://api.memorysync.io/v1/memory/add_turn \--header "X-API-Key: $MEMORYSYNC_API_KEY" \--header "Content-Type: application/json" \--data '{"tenant_id":"acme","user_id":"caller-42","source":"pipecat","text":"book the same hotel as last time","role":"user","speaker":"human@pipecat::call-123#h<content-hash>","metadata":{"session_id":"pipecat::call-123"}}'
Placement matters: after context_aggregator.user() so the frame carries the full context, before llm so enrichment reaches the model. The service is reusable across pipeline runs; call await memory.aclose() at application shutdown for an explicit flush.
Complete runnable example
import osfrom dotenv import load_dotenvfrom pipecat.audio.vad.silero import SileroVADAnalyzerfrom pipecat.pipeline.pipeline import Pipelinefrom pipecat.pipeline.runner import PipelineRunnerfrom pipecat.pipeline.task import PipelineTaskfrom pipecat.services.cartesia.tts import CartesiaTTSServicefrom pipecat.services.deepgram.stt import DeepgramSTTServicefrom pipecat.services.openai.llm import OpenAILLMServicefrom pipecat.processors.aggregators.llm_context import LLMContextAggregatorPairfrom pipecat.runner.types import RunnerArgumentsfrom pipecat.transports.base_transport import TransportParamsfrom pipecat_memorysync import MemorySyncMemoryServiceload_dotenv()async def bot(runner_args: RunnerArguments):stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))llm = OpenAILLMService(model="your-model")tts = CartesiaTTSService(api_key=os.getenv("CARTESIA_API_KEY"))memory = MemorySyncMemoryService(user_id=os.getenv("MEMORYSYNC_USER_ID", "caller-42"),session_id="call-123",)context = LLMContextAggregatorPair()transport = ... # your transport (see any Pipecat foundational example)pipeline = Pipeline([transport.input(),stt,context.user(),memory, # ← the only new linellm,tts,transport.output(),context.assistant(),])runner = PipelineRunner()await runner.run(PipelineTask(pipeline))
Say "My name is Alex and my favorite color is teal", disconnect, refresh the page, reconnect — the bot greets you with what it remembers. The processor is one added line in any of Pipecat’s foundational examples; everything else is a stock pipeline.
Understanding user_id and session_id
| Field | What it scopes | Choose it like this |
|---|---|---|
user_id | The person. All recall and all facts attach to this id — it is what makes the caller recognized across calls and across every MemorySync integration. | A stable id from YOUR system: account id, CRM id, or the caller’s number. Never a random per-run value. |
session_id | The conversation. Groups the facts distilled from this call under one session scope (pipecat::<session>). | The call or room id. Optional — omit it and the facts still store under the user. |
What runs when
| Moment | What happens |
|---|---|
LLMContextFrame arrives | Recall runs under the budget; relevant memories land as a system message (position configurable). The frame is pushed on time either way. |
| Same frame, before it is pushed on | New user messages — and only new ones — are queued in the background for fact extraction. Assistant replies are not sent, and the injected memory block is excluded. |
EndFrame (graceful end) | Queued sends get a bounded 3s window to land before the pipeline stops — the caller’s last words are not lost. |
CancelFrame (abort) | The frame is pushed first; sends get a brief salvage window. |
| Backend slow / dead / over quota | Unenriched frame, silent skip — the pipeline is never affected. Free-tier quota exhaustion is silent by design; evaluation keys surface strict 429s instead. |
Configuration
| Parameter | Default | Meaning |
|---|---|---|
top_k | 6 | Memories injected per turn |
recall_timeout | 1.2 | Hard recall budget in seconds |
system_prompt | guarded header | Prefix line for the injected block; doubles as the capture-exclusion marker |
add_as_system_message | True | Inject as a system message (else as a user message) |
position | 1 | Index in the message list where the memory block is inserted (0 puts it first; the default lands right after your system prompt) |
min_prompt_chars | 8 | Skip recall for trivial prompts |
Example: what the caller experiences
| Call 1 — caller | "Hi! My name is Alex and my favorite color is teal." |
| Bot | "Great to meet you, Alex! Teal it is. What can I do for you?" |
| *(server, seconds later)* | Extracted facts land: *Name is Alex* · *Favorite color is teal* — not the caller’s words, no "ok"/"hmm" junk, nothing from the bot’s reply |
| Call 2 — caller | "Do you remember anything about me?" |
| Bot | "Of course — you’re Alex, and your favorite color is teal!" |
What happens server-side
| Concern | Behaviour |
|---|---|
| What is stored | The caller’s turns go to fact extraction, and only the durable facts in them are stored — never the turn text, and nothing from the assistant’s replies. It is the pipeline every MemorySync integration shares. Speech-pause fragments ("and", "dinner") and filler turns store nothing. |
| Capture timing | Every new user message is in flight the moment its frame passes — BEFORE the LLM replies — so a browser disconnect can never lose the caller’s last (usually most important) sentence. |
| Billing | Each sent user turn counts one add request. Each enrichment is a recall: one retrieval request, plus one more when it returns no context and the plain semantic query fallback runs. A frame repeating the last user message is not recalled again. Deletes are free. |
| Over quota | Production keys degrade silently — sends are accepted and skipped (processing_status: "skipped"), recalls answer empty — a billing state is never spoken to a caller. Evaluation keys get a strict 429 instead. |
| Duplicates | Deterministic idempotency seeds — reconnects and frame replays are recognised server-side and extracted once. |
Verify it is working
# list what the bot remembers about a callercurl "https://api.memorysync.io/v1/memory/acme/caller-42/list?limit=20" \--header "X-API-Key: $MEMORYSYNC_API_KEY"
Or open Memory Explorer in the dashboard and filter by the user id — facts appear within seconds of each turn (extraction is asynchronous, typically under 10s).
Troubleshooting
| Symptom | Cause and fix |
|---|---|
| Bot never remembers anything | Almost always a user_id that changes between runs — pin it. Second check: the key needs write scope. |
| Remembers within a call, not across calls | Extraction is asynchronous — give it ~10s after the call before reconnecting. If facts exist in Memory Explorer but recall misses them, raise top_k. |
| Replies feel slow on a home network | Raise recall_timeout (e.g. 6.0) during development — the 1.2s default is tuned for datacenter latency and skips injection rather than wait. |
| Reconnect hangs in the local playground | Refresh the page (F5) between disconnect and reconnect — a quirk of the dev client, not of the pipeline or the memory service. |
| Nothing stores and nothing errors | Quota exhausted — production keys degrade silently by design. Check the Usage page. |
Privacy and data safety
Only extracted facts persist — the caller’s turns are processed and discarded, and assistant replies are never sent, so chit-chat and mishears never become permanent records. The injected memory block is capture-excluded and can never re-enter storage. Deletion is a first-class API (/memory/forget) and dashboard action, per end user. The memory engine sees text only — audio never reaches MemorySync.
Supported versions
| Surface | Requires | Verified on |
|---|---|---|
pipecat-memorysync 1.2.1 | pipecat-ai 1.0+, Python 3.10+ | 19 CI checks driven through pipecat.tests.utils.run_test — the framework’s official harness — on the latest pipecat-ai: budget enforcement, immediate loss-proof capture (including the exact disconnect sequence), delta-only proof, injection exclusion, user-turn-only sending (replies never sent), quota modes |