Amazon S3 integration
Index documents from the S3 bucket paths an administrator approves, including S3-compatible stores such as Cloudflare R2, MinIO, Wasabi, and Backblaze B2. Retrieve the contents of documents held in object storage, with the bucket, key, and a link back to the object.
What becomes retrievable
Retrieve the contents of documents held in object storage, with the bucket, key, and a link back to the object.
Plain text, Markdown, CSV, JSON, and other text formats.
PDF, Word, PowerPoint, and Excel files, with text extracted.
AWS S3 plus S3-compatible endpoints via a custom endpoint URL.
Connecting with IAM credentials
S3 is storage, not an identity provider, so there is no sign-in redirect. An administrator creates a read-only IAM user for the bucket and enters its access key. MemorySync proves the key can both locate the bucket and list its contents before the connection is saved, so an over-narrow policy fails on the connect screen rather than as a sync that quietly returns nothing.
- An S3 bucket, or a bucket on an S3-compatible service, holding documents worth retrieving.
- A read-only IAM user with s3:ListBucket on the bucket and s3:GetObject on its contents. Add kms:Decrypt if the bucket uses SSE-KMS.
- A MemorySync project for the imported documents.
Choose path scope
What is downloaded, and what is only listed
Every exclusion is decided from the listing entry alone — key, size, storage class, and ETag — before any object is fetched. That ordering is deliberate: S3 bills requests and egress to your account, so an object that will not be indexed costs nothing but the listing it already appeared in.
| Decision | Made from | Cost |
|---|---|---|
| Excluded by pattern | The key | No download |
| Unsupported type | The key’s extension | No download |
| In Glacier or Deep Archive | The storage class | No download |
| Over the size limit | The reported size | No download |
| Unchanged since last sync | The ETag | No download |
| Indexed | All gates passed | One read |
S3-compatible services
Anything speaking the S3 API works by supplying a custom endpoint on the connect screen: Cloudflare R2, MinIO, Wasabi, Backblaze B2, and others. Non-AWS endpoints use path-style addressing automatically, which is what most compatible services expect.
Guided first connection checklist
Check items for your own planning. Nothing here changes a live connection.
0 of 4 planning steps complete
Initial and incremental ingestion
- 1Add IAM credentials
An administrator enters a read-only access key, which is verified against the bucket before it is saved.
- 2Approve paths
They select each path to index; approving a path covers everything beneath it.
- 3Index readable files
MemorySync lists each approved path and downloads only files it can read.
- 4Sync on a schedule
Later runs compare each object’s ETag, so unchanged files are never downloaded again.
Verify with a real document question
Query a distinctive phrase from a document you know sits under an approved path. Confirm the result names the right bucket and key before widening path access.
Troubleshooting and related sources
Why were the credentials rejected?
MemorySync checks both that the bucket exists and that its contents can be listed. A key that can list but not read will connect and then index nothing, so confirm the policy grants s3:GetObject on the bucket’s contents as well as s3:ListBucket on the bucket itself.
Why did only a few files index out of thousands?
That is usually correct. Buckets are full of things that are not documents, and each approved path reports what it skipped and why: unsupported types, archived storage classes, oversized files, and keys matched by an exclusion pattern.
Why does listing work but every download fail with access denied?
The bucket is almost certainly encrypted with SSE-KMS and the IAM policy is missing kms:Decrypt. S3 reports this as a plain access denial on GetObject, which makes it look like an S3 permission problem rather than a key policy one.
Why is a deleted file still retrievable?
S3 sends no deletion notifications, so removals are detected by comparing a fresh listing against what is stored. That happens on a periodic reconciliation rather than immediately.
Does connecting a bucket cost anything on my AWS account?
Listing and reading objects are billed to you as requests and egress. MemorySync is built to keep that small: every exclusion is decided from the listing entry, so a file that will not be indexed is never downloaded, and unchanged files are skipped by ETag comparison rather than re-read.