Web Crawler

Operations on this page

Operation	Method and path	Python	Node.js
Read crawler limits	`GET …/web-crawler/config`	`integrations.web_crawler.config`	`integrations.webCrawler.config`
Validate a URL	`POST …/web-crawler/validate`	`integrations.web_crawler.validate`	`integrations.webCrawler.validate`
Start a crawl	`POST …/web-crawler/crawl`	`integrations.web_crawler.crawl`	`integrations.webCrawler.crawl`
List crawl jobs	`GET …/web-crawler/jobs`	`integrations.web_crawler.jobs`	`integrations.webCrawler.jobs`
Get one job	`GET …/web-crawler/jobs/{job_id}`	`integrations.web_crawler.job`	`integrations.webCrawler.job`
List running crawls	`GET …/web-crawler/active`	`integrations.web_crawler.active`	`integrations.webCrawler.active`
Read fetched pages	`GET …/web-crawler/jobs/{job_id}/content`	`integrations.web_crawler.job_content`	`integrations.webCrawler.jobContent`
Read job statistics	`GET …/web-crawler/jobs/{job_id}/statistics`	`integrations.web_crawler.job_statistics`	`integrations.webCrawler.jobStatistics`
Import pages as memories	`POST …/web-crawler/jobs/{job_id}/import`	`integrations.web_crawler.import_job`	`integrations.webCrawler.importJob`
Cancel a crawl	`POST …/web-crawler/jobs/{job_id}/cancel`	`integrations.web_crawler.cancel_job`	`integrations.webCrawler.cancelJob`
Delete a job	`DELETE …/web-crawler/jobs/{job_id}`	`integrations.web_crawler.delete_job`	`integrations.webCrawler.deleteJob`

Authentication and scope

Requirement	Contract
Credential	An API key sent as `X-API-Key`. Connector operations are not end-user scoped.
Read scope	`integrations:read` for every `GET`.
Write scope	`integrations:write` for every `POST`, `PUT`, `PATCH` and `DELETE`.
Tenant	Derived from the authenticated key. There is no tenant parameter to pass or to get wrong.
`X-End-User-ID`	Not used. A connection belongs to the organization, not to one end user.

There is no connection to create

Unlike Slack or S3 there is no provider to authorise and no connection_id. A crawl targets a public URL, so the crawler is reached directly under /api/v1/integrations/web-crawler, and jobs are scoped to your organization by the API key alone.

How a crawl becomes memories

Two stages, one decision point

11. Validate
Check the URL is reachable and permitted before spending a job on it.
22. Crawl
Pages are fetched and stored. No memories exist yet.
33. Inspect
Read the fetched pages and the statistics. Decide what is worth keeping.
44. Import
The pages you choose become memories. This is the step that changes your memory store.
55. Tidy up
Cancel a crawl that is going wrong; delete a finished job you no longer need.

1. Read the limits before designing a crawl

The ceilings are set by the deployment, not by your request. Reading them first means a client can present real bounds instead of guessing and being rejected.

import os
from memorysync import MemorySyncClient

client = MemorySyncClient(
    api_key=os.environ["MEMORYSYNC_API_KEY"],
    base_url="https://api.memorysync.io",
)

limits = client.integrations.web_crawler.config()
print(limits["max_depth"], limits["max_pages"])
print(limits["allowed_protocols"], limits["rate_limit_ms"], "ms between requests")

GET/api/v1/integrations/web-crawler/config

200 OK

Field	Meaning
`max_depth`	The deepest link depth the deployment permits.
`max_pages`	The most pages one crawl may fetch.
`rate_limit_ms`	Delay the crawler leaves between requests to one host.
`max_content_size_bytes`	Per-page ceiling. A larger page is skipped rather than truncated.
`max_total_crawl_size_bytes`	Ceiling across the whole job.
`max_crawl_duration_minutes`	Wall-clock ceiling. A crawl that exceeds it stops where it got to.
`allowed_protocols`	Schemes the crawler will follow.

2. Validate the URL first

Validation is cheap and tells you whether a crawl is worth starting. It checks reachability and whether the URL is permitted, and returns the domain it resolved.

import os
from memorysync import MemorySyncClient

client = MemorySyncClient(
    api_key=os.environ["MEMORYSYNC_API_KEY"],
    base_url="https://api.memorysync.io",
)

check = client.integrations.web_crawler.validate("https://example.com/handbook")
if not check["valid"]:
    print("cannot crawl:", check["error_code"], check["error"])
else:
    print("will crawl", check["domain"])

POST/api/v1/integrations/web-crawler/validate

200 OK

3. Start the crawl

A crawl returns a job immediately. The depth is the setting that decides whether this is one page or a site: max_depth of 0 fetches only the URL you gave.

import os
from memorysync import MemorySyncClient

client = MemorySyncClient(
    api_key=os.environ["MEMORYSYNC_API_KEY"],
    base_url="https://api.memorysync.io",
)

job = client.integrations.web_crawler.crawl(
    "https://example.com/handbook",
    crawl_type="page",
    settings={"max_depth": 1, "max_pages": 25, "store_html": False},
)
print(job["id"], job["status"], job["domain"])

POST/api/v1/integrations/web-crawler/crawl

201 Created

Request field	Contract
`url`	Required. The page to start from.
`crawl_type`	Optional, defaults to `page`. Also accepts `article` and `document`.
`settings.max_depth`	Optional, `0`–`5`, defaults to `0`. Zero means the single page you named.
`settings.max_pages`	Optional, `1`–`1000`, defaults to `10`.
`settings.store_html`	Optional, defaults to `false`. Keeps the original markup as well as the text.
`settings.include_images`	Optional, defaults to `false`. Records image URLs in page metadata.

4. Watch the job

Poll the job, or list the ones running now. can_import is the flag worth branching on — it tells you the job reached a state where importing is possible, without your having to interpret the status string.

import os
from memorysync import MemorySyncClient

client = MemorySyncClient(
    api_key=os.environ["MEMORYSYNC_API_KEY"],
    base_url="https://api.memorysync.io",
)

job = client.integrations.web_crawler.job("crawl_5d90e2")
print(job["status"], job["pages_crawled"], "of", job["pages_found"])
print("failed:", job["pages_failed"], "skipped:", job["pages_skipped"])
if job["can_import"]:
    print("ready to import")

for running in client.integrations.web_crawler.active():
    print("in flight:", running["id"], running["url"])

GET/api/v1/integrations/web-crawler/jobs/{job_id}

200 OK

GET/api/v1/integrations/web-crawler/active

200 OK

Job field	Meaning
`id`, `url`, `domain`, `crawl_type`	What was asked for.
`status`	Where the job is. `pending` and `processing` mean still running.
`pages_found`, `pages_crawled`	Discovered versus successfully fetched.
`pages_failed`, `pages_skipped`	Fetch errors, and pages excluded by size or type.
`error_message`, `error_code`	Why the job itself failed, when it did.
`created_at`, `started_at`, `completed_at`	Timestamps. Null until each stage is reached.
`can_import`	Whether the fetched pages can be imported now.

5. List and page through jobs

The job list is paginated with limit and offset, and filtered with status.

import os
from memorysync import MemorySyncClient

client = MemorySyncClient(
    api_key=os.environ["MEMORYSYNC_API_KEY"],
    base_url="https://api.memorysync.io",
)

page = client.integrations.web_crawler.jobs(status="completed", limit=20, offset=0)
print(page["total"], "jobs")
for job in page["items"]:
    print(job["id"], job["url"], job["pages_crawled"])

GET/api/v1/integrations/web-crawler/jobs

200 OK

Query parameter	Contract
`status`	Optional. Filter to one job status.
`limit`	Optional, `1`–`100`, defaults to `20`.
`offset`	Optional, `0` or more, defaults to `0`.

6. Look at what was fetched, before importing

This is the step that makes the two-stage design worth having. Read the pages the crawl stored and decide which are worth keeping — a navigation page or a login wall is not.

import os
from memorysync import MemorySyncClient

client = MemorySyncClient(
    api_key=os.environ["MEMORYSYNC_API_KEY"],
    base_url="https://api.memorysync.io",
)

pages = client.integrations.web_crawler.job_content("crawl_5d90e2", limit=50, offset=0)
print(pages["total"], "pages fetched")
for content in pages["items"]:
    print(content["id"], content["status_code"], content["content_length"], content["title"])
    print("  ", content["url"])
    print("  ", (content["content_preview"] or "")[:120])

stats = client.integrations.web_crawler.job_statistics("crawl_5d90e2")
print(stats["pages_crawled"], "crawled in", stats["crawl_duration_seconds"], "s")
print("average", stats["average_page_time_ms"], "ms per page")

GET/api/v1/integrations/web-crawler/jobs/{job_id}/content

200 OK

GET/api/v1/integrations/web-crawler/jobs/{job_id}/statistics

200 OK

Page field	Meaning
`id`	The content id. Pass these to import to select specific pages.
`url`, `title`, `depth`	Where it came from and how far in.
`content_preview`, `content_length`	The start of the extracted text, and its full length.
`status`, `status_code`, `error_message`	Fetch outcome, including the HTTP status the site returned.
`content_size_bytes`	Stored size, useful for spotting a page that hit the size ceiling.
`metadata`	Extracted metadata, including image URLs when `include_images` was set.
`crawled_at`, `crawl_duration_ms`	When it was fetched and how long it took.
`memory_id`, `imported_at`	Set once the page has been imported. Null means not yet imported.

Statistic	Meaning
`pages_crawled`, `pages_failed`, `pages_skipped`	Outcome counts for the job.
`total_content_bytes`, `total_html_bytes`	Extracted text and, when stored, original markup.
`average_page_time_ms`, `crawl_duration_seconds`	How long it took.

7. Import the pages worth keeping

Importing is what creates memories. Omit content_ids to import everything the job fetched, or pass the specific ids you chose in the previous step. Tags are applied to every memory the import creates, which is what makes a crawl's output findable and removable as a set.

import os
from memorysync import MemorySyncClient

client = MemorySyncClient(
    api_key=os.environ["MEMORYSYNC_API_KEY"],
    base_url="https://api.memorysync.io",
)

result = client.integrations.web_crawler.import_job(
    "crawl_5d90e2",
    content_ids=["cnt_4a1", "cnt_4a2"],
    tags=["handbook", "crawl-2026-08"],
)
print(result["imported_count"], "imported,", result["failed_count"], "failed")
print(result["memory_ids"])
for error in result["errors"]:
    print("error:", error)

POST/api/v1/integrations/web-crawler/jobs/{job_id}/import

200 OK

Request field	Contract
`content_ids`	Optional. Specific page ids to import. Omit to import everything the job fetched.
`tags`	Optional. Applied to every memory this import creates.

Response field	Meaning
`success`	Whether the import completed as a whole.
`imported_count`, `failed_count`	Pages turned into memories, and pages that could not be.
`memory_ids`	The memories created. Keep these if you may need to remove the set later.
`errors`	Per-page reasons. Read these rather than only the counts.

8. Cancel a crawl, or delete a finished job

Cancelling stops a running crawl and keeps the pages already fetched, so a crawl that is going wider than intended can be stopped and still be useful. Deleting removes the job and its fetched pages.

import os
from memorysync import MemorySyncClient

client = MemorySyncClient(
    api_key=os.environ["MEMORYSYNC_API_KEY"],
    base_url="https://api.memorysync.io",
)

client.integrations.web_crawler.cancel_job("crawl_5d90e2")

# Only once it is no longer running.
client.integrations.web_crawler.delete_job("crawl_5d90e2")

POST/api/v1/integrations/web-crawler/jobs/{job_id}/cancel

200 OK

DELETE/api/v1/integrations/web-crawler/jobs/{job_id}

200 OK

Errors and next action

Status	Meaning	Next action
`401`	Missing, malformed or inactive API key.	Check server configuration without printing the key.
`403`	The key lacks `integrations:read` or `integrations:write`.	Grant the scope on the key, or use a key that has it.
`404`	The connection, object or job is not visible to this tenant.	Confirm the identifier belongs to this organization.
`409`	The connection is in a state that forbids the operation.	Read the connection status first and act on it.
`429`	Rate limited, either by MemorySync or by the upstream provider.	Back off; do not tighten a polling loop in response.
`5xx`	Service failure.	Treat a write outcome as uncertain and reconcile by reading the connection back.

Upload File

Send a document directly instead of crawling for it.

Legacy Integrations

The v1 catalog this connector is listed in.

Forget

Remove memories an import created.

Operations on this page

Authentication and scope

There is no connection to create

How a crawl becomes memories

1. Read the limits before designing a crawl

2. Validate the URL first

3. Start the crawl

4. Watch the job

5. List and page through jobs

6. Look at what was fetched, before importing

7. Import the pages worth keeping

8. Cancel a crawl, or delete a finished job

Errors and next action

Related reference