Docs
API ReferenceReferenceKnowledgebases

Crawl a website

POST
/knowledgebases/{knowledgebaseId}/crawls

Crawls a website into this knowledgebase: a sitemap, a whole site, or a section (a start URL like https://acme.com/docs with no include crawls /docs). Pages come from the sitemap (the one given, robots.txt, or /sitemap.xml), else by following same-site links; robots.txt is respected. Each page becomes a source. Send preview: true first: it answers pages found, the room left under the plan's sources-per-knowledgebase cap, and estimatedCredits, with nothing written. Each page fetched and indexed costs what one Web Scrape costs (creditsPerPage). A new crawl needs credits: with none left the answer is 402 insufficient_credits, and a run that runs out partway stops, keeps every page it has, and resumes once credits are back. Otherwise the answer is 202 with the crawl; pages arrive in the background, 50 a run; follow with GET. The same site into the same knowledgebase again updates that crawl. Requires the knowledge:write scope and an org key or an owner or admin personal key.

AuthorizationBearer <token>

An organization API key, created from a Portal's Webhooks page (Provisioning API card). The secret is shown once at creation and stored only as a hash. Keys are organization-wide and carry scopes (end_users:read, end_users:write). Rate limit: 120 requests per minute per key (sliding 60-second window).

In: header

Path Parameters

knowledgebaseId*string
Formatuuid

Request Body

application/json

TypeScript Definitions

Use the request body type in TypeScript.

Response Body

application/json

application/json

application/json

application/json

application/json

application/json

application/json

application/json

application/json

curl -X POST "https://example.com/knowledgebases/497f6eca-6276-4993-bfeb-53cbbbba6f08/crawls" \  -H "Content-Type: application/json" \  -d '{    "url": "string"  }'
{  "preview": {    "startUrl": "string",    "mode": "sitemap",    "found": 0,    "pages": 0,    "room": 0,    "willImport": 0,    "overCap": 0,    "creditsPerPage": 0,    "estimatedCredits": 0,    "partial": true,    "notes": [      "string"    ],    "sample": [      "string"    ],    "sections": [      {        "path": "string",        "pages": 0      }    ]  }}