Crawl a website
Crawls a website into this knowledgebase: a sitemap, a whole site, or a section (a start URL like https://acme.com/docs with no include crawls /docs). Pages come from the sitemap (the one given, robots.txt, or /sitemap.xml), else by following same-site links; robots.txt is respected. Each page becomes a source. Send preview: true first: it answers pages found, the room left under the plan's sources-per-knowledgebase cap, and estimatedCredits, with nothing written. Each page fetched and indexed costs what one Web Scrape costs (creditsPerPage). A new crawl needs credits: with none left the answer is 402 insufficient_credits, and a run that runs out partway stops, keeps every page it has, and resumes once credits are back. Otherwise the answer is 202 with the crawl; pages arrive in the background, 50 a run; follow with GET. The same site into the same knowledgebase again updates that crawl. Requires the knowledge:write scope and an org key or an owner or admin personal key.
Authorization
apiKey An organization API key, created from a Portal's Webhooks page (Provisioning API card). The secret is shown once at creation and stored only as a hash. Keys are organization-wide and carry scopes (end_users:read, end_users:write). Rate limit: 120 requests per minute per key (sliding 60-second window).
In: header
Path Parameters
uuidRequest Body
application/json
TypeScript Definitions
Use the request body type in TypeScript.
Response Body
application/json
application/json
application/json
application/json
application/json
application/json
application/json
application/json
application/json
curl -X POST "https://example.com/knowledgebases/497f6eca-6276-4993-bfeb-53cbbbba6f08/crawls" \ -H "Content-Type: application/json" \ -d '{ "url": "string" }'{ "preview": { "startUrl": "string", "mode": "sitemap", "found": 0, "pages": 0, "room": 0, "willImport": 0, "overCap": 0, "creditsPerPage": 0, "estimatedCredits": 0, "partial": true, "notes": [ "string" ], "sample": [ "string" ], "sections": [ { "path": "string", "pages": 0 } ] }}