Amazon S3 integration

What becomes retrievable

Retrieve the contents of documents held in object storage, with the bucket, key, and a link back to the object.

Connecting with IAM credentials

S3 is storage, not an identity provider, so there is no sign-in redirect. An administrator creates a read-only IAM user for the bucket and enters its access key. MemorySync proves the key can both locate the bucket and list its contents before the connection is saved, so an over-narrow policy fails on the connect screen rather than as a sync that quietly returns nothing.

• An S3 bucket, or a bucket on an S3-compatible service, holding documents worth retrieving.
• A read-only IAM user with s3:ListBucket on the bucket and s3:GetObject on its contents. Add kms:Decrypt if the bucket uses SSE-KMS.
• A MemorySync project for the imported documents.

Choose path scope

What is downloaded, and what is only listed

Every exclusion is decided from the listing entry alone — key, size, storage class, and ETag — before any object is fetched. That ordering is deliberate: S3 bills requests and egress to your account, so an object that will not be indexed costs nothing but the listing it already appeared in.

S3-compatible services

Anything speaking the S3 API works by supplying a custom endpoint on the connect screen: Cloudflare R2, MinIO, Wasabi, Backblaze B2, and others. Non-AWS endpoints use path-style addressing automatically, which is what most compatible services expect.

Guided first connection checklist

Initial and incremental ingestion

1. 1
   Add IAM credentials

   An administrator enters a read-only access key, which is verified against the bucket before it is saved.

2. 2
   Approve paths

   They select each path to index; approving a path covers everything beneath it.

3. 3
   Index readable files

   MemorySync lists each approved path and downloads only files it can read.

4. 4
   Sync on a schedule

   Later runs compare each object’s ETag, so unchanged files are never downloaded again.

Verify with a real document question

Query a distinctive phrase from a document you know sits under an approved path. Confirm the result names the right bucket and key before widening path access.

Troubleshooting and related sources

Operations on this page

Authentication and scope

Before you start

S3 is the credential-based connector: create it with create_with_credentials on Connection Lifecycle, passing access_key_id, secret_access_key, region and bucket. There is no OAuth flow.

1. See what the credentials can reach

Lists prefixes visible in the bound bucket, annotated with whether each is already approved.

import os
from memorysync import MemorySyncClient

client = MemorySyncClient(
    api_key=os.environ["MEMORYSYNC_API_KEY"],
    base_url="https://api.memorysync.io",
)

available = client.connections.s3.available_prefixes("conn_8f21a4")
print(available["bucket"], available["total"])
for entry in available["prefixes"]:
    print(entry["prefix"] or "(root)", entry["kind"], entry["approved"])

2. Approve the prefixes to read

Bare strings are accepted for the common case, and objects when you want to carry a label. Approving a prefix starts the first sync straight away if the connection is healthy.

import os
from memorysync import MemorySyncClient

client = MemorySyncClient(
    api_key=os.environ["MEMORYSYNC_API_KEY"],
    base_url="https://api.memorysync.io",
)

result = client.connections.s3.add_prefixes(
    "conn_8f21a4",
    ["handbook/", "policies/2026/"],
)
print(result["sync_triggered"])
for approved in result["approved"]:
    print(approved["prefix"], approved["state"])
for rejected in result["rejected"]:
    print("rejected:", rejected)

3. Read back what is approved

Another of the bare-array responses: iterate the result itself rather than looking for a prefixes key.

import os
from memorysync import MemorySyncClient

client = MemorySyncClient(
    api_key=os.environ["MEMORYSYNC_API_KEY"],
    base_url="https://api.memorysync.io",
)

for row in client.connections.s3.prefixes("conn_8f21a4"):
    print(row["prefix"] or "(root)", row["state"], row["is_active"])
    print("  synced:", row["objects_synced"], "indexed:", row["objects_indexed"])
    print("  skipped:", row["skipped"])

4. Block keys by pattern

The exclusion policy is a list of glob patterns matched against keys. A pattern blocks matching keys inside every approved prefix, which is how you keep a useful prefix without importing its build artifacts or backups.

import os
from memorysync import MemorySyncClient

client = MemorySyncClient(
    api_key=os.environ["MEMORYSYNC_API_KEY"],
    base_url="https://api.memorysync.io",
)

client.connections.s3.set_exclusion_policy(
    "conn_8f21a4",
    tenant_patterns=["*.log", "backups/*", "*/tmp/*"],
)
policy = client.connections.s3.exclusion_policy("conn_8f21a4")
print(policy["effective_patterns"])
print("removed:", policy["purged_objects"], "objects,", policy["purged_memories"], "memories")

5. Read the limits that decide what is parsed

Effective settings, including the ceilings that quietly exclude objects. An object over the size limit or with an unlisted extension produces no memories and no error, so this is the endpoint that explains an unexpectedly empty prefix.

import os
from memorysync import MemorySyncClient

client = MemorySyncClient(
    api_key=os.environ["MEMORYSYNC_API_KEY"],
    base_url="https://api.memorysync.io",
)

settings = client.connections.s3.settings("conn_8f21a4")
print(settings["bucket"], settings["region"], settings["endpoint"])
print("max object MB:", settings["max_object_mb"])
print("max objects per sync:", settings["max_objects_per_sync"])
print("allowed extensions:", settings["allowed_extensions"])

6. Revoke one prefix

Removes a single approval. The prefix travels as a query parameter, not in the body and not as a path segment, because prefixes contain slashes.

import os
from memorysync import MemorySyncClient

client = MemorySyncClient(
    api_key=os.environ["MEMORYSYNC_API_KEY"],
    base_url="https://api.memorysync.io",
)

# Stop reading it, keep what it already produced.
client.connections.s3.remove_prefix("conn_8f21a4", "policies/2026/")

# Stop reading it and delete the memories it produced.
client.connections.s3.remove_prefix("conn_8f21a4", "policies/2026/", purge=True)

Errors and next action