How It Works

Background Jobs

Anything that does not have to happen during a request happens in the background. There are two background mechanisms in MemorySync — a scheduler that fires recurring jobs at fixed cadences, and a task queue that runs ad-hoc work enqueued by request handlers. This page lists every recurring job, what triggers it, and how to recognise its output.

Two mechanisms, not one

Mechanism	Triggered by	Examples
Scheduler	Wall clock — fixed cadences (every 5 minutes, every hour, daily, weekly).	Auto-summarization, retention enforcement, tier transitions, scheduled integration syncs, audit log retention, daily insights generation.
Task queue	A request handler or another task enqueues an async job.	Webhook delivery, SIEM forwarding, intelligence re-evaluation kicked off by recall, async embedding refresh.

Every recurring job, what it does, and how often

Job	Cadence	What it does
`auto_summarization_job`	Periodic (defense in depth)	Picks up clusters whose member count crossed the summary threshold but whose inline summary did not run, and produces the summary. Most summaries are generated inline at write time — see Summarization Pipeline.
`apply_forgetting_policies_job`	Hourly	Decays unused memories — drops `decay_score` on rows with no recall in the configured window.
`tier_transition_job`	Hourly	Moves memories between `hot`, `warm`, and `cold` based on access patterns.
`retention_enforcement_job`	Daily	Walks rows whose `retention_expires_at` is in the past and progresses them through the lifecycle (`active` → `archived` → `soft_deleted` → `hard_delete_pending` → `purged`).
`generate_daily_insights_job`	Daily	Recomputes per-tenant lifetime intelligence snapshots using bounded sampling.
`process_capture_queue_job`	Every 5 min	Drains pending memory candidates produced by integrations and promotes the validated ones into the `memories` table.
`process_scheduled_syncs`	Every 5 min	Discovers integration connections whose `next_scheduled_sync` is due and dispatches a sync job for each.
`siem_forwarding_job`	Every 5 min	Reads new audit log rows and forwards them to each tenant's configured SIEM endpoint in batches of 100.
`cleanup_expired_sessions`	Hourly	Removes expired refresh-token sessions from the session table.
`intelligence_reevaluate`	Every 10 min	Scans recall events for memories that became hot since their last evaluation and re-scores them. See Re-evaluation Flow.

How the scheduler stays honest under restarts

Single-instance enforcement: every recurring job has a single-instance lock — a job cannot overlap with itself even if the previous run is still finishing.
Coalesce on miss: if the scheduler was down for an hour and a 5-minute job missed twelve fires, it runs once on resume, not twelve times.
Single dedicated process: the scheduler runs in one process, not per worker. Two API workers will never both fire the same cron.
UTC throughout: every cadence is anchored to UTC so daylight savings does not shift jobs.

How tasks retry when they fail

Every task is registered with automatic retries. A failed task is re-enqueued with exponential backoff up to a per-task ceiling. After the ceiling, the task is moved to the dead-letter queue and a structured log line is emitted. Critical paths (SIEM forwarding, webhook delivery) carry their own retry schedules — see Failure Modes & Retries.

What running jobs look like in the logs

TEXT

[scheduler] tick: job=auto_summarization_job started at=2026-05-04T10:00:00Z
[scheduler] tick: job=auto_summarization_job finished
             scanned=842 summarised=37 skipped_unchanged=805 duration_ms=14322

[task] webhook.deliver id=wd_8821 attempt=1 status=success http_status=200
[task] webhook.deliver id=wd_8822 attempt=1 status=retry  http_status=503
[task] webhook.deliver id=wd_8822 attempt=2 status=retry  http_status=503
[task] webhook.deliver id=wd_8822 attempt=3 status=dead_letter

What to watch when a background flow is stuck

Queue depth metric — if it grows linearly without recovering, the workers are saturated; add workers, do not raise per-worker concurrency.
Last-success timestamp per job — every recurring job records the timestamp of its last successful tick. If the timestamp is more than two cadences old, the job has been failing.
Dead-letter table — every promoted dead-letter row carries the original payload and the last error. Fix the root cause, then replay.

← Previous

Request Lifecycle

Caching Layers