Website Crawls
Crawl a whole website, or one section of it, into a Knowledgebase and keep it current on a schedule.
Website Crawls
A website crawl reads every page of a site, or of one section like /docs, into a Knowledgebase. Each page becomes a File your Agents and Flows can search. On a schedule, FormWise checks the site again and brings in new and changed pages, so a help-center or docs chatbot answers from what the site says today.
For a single page, add it as a regular File instead: Add a link, then This page.
Starting a crawl
- Open the Knowledgebase, open the New menu and choose Add a link, then switch to Whole site. On an empty Knowledgebase, click Add a website.
- Enter an Address: a site (
https://docs.example.com), a section of one (https://example.com/docs), or a sitemap (https://example.com/sitemap.xml). - Click Check site. FormWise finds the site's pages and shows them as a tree, with nothing written yet.
- Tick the sections and pages you want. Ticking a section includes everything under it, including pages added to it later.
- Set the Page limit (200 by default, up to 1,000) and how often to Check again: Manual, Weekly, Daily or Hourly.
- Check the summary: pages ticked, pages over the page limit or over your plan, how many pages it Will crawl, and about how many Credits that costs. Then click Crawl N pages.
The crawl arrives as a folder in your Knowledgebase, named after the site, and its pages fill in as they are read. Each File is named after the page's title and starts with the address it came from, so answers can cite it.
How pages are found
- Sitemap first. FormWise reads the sitemap you gave, the ones the site's
robots.txtlists, or/sitemap.xml. Sitemap indexes and gzipped sitemaps work. - Otherwise, links. With no sitemap, it follows links on the site from the start page, up to three links deep.
- Same site only. Pages on other domains are never read.
www.andhttpversushttpscount as the same site. - The site's rules are respected. Pages that
robots.txtblocks for FormWiseBot, and pages markednoindex, are left out. Private and internal network addresses are refused. - Only the content. Navigation, headers, footers, cookie banners and sidebars are left out of each page.
What it costs
A crawled page costs the same as the Web Scrape action an Agent uses: 1 credit per page fetched and indexed. Pages that haven't changed cost nothing.
- Before you start, the summary shows the estimate for the pages it will crawl.
- For a schedule, it shows the most the schedule can cost if every page changed every time: for example, "Daily checks cost up to 180 credits a day. Only new and changed pages are charged." The real number is usually much lower.
- After each run, the crawl's row in the Knowledgebase shows the credits that run used.
Out of credits: a new crawl can't start. A crawl that runs out partway stops within a few pages, keeps every page it already has, and says so on its row. It picks up again at its next scheduled check, or when you click Sync now, once credits are back. Running out of credits never deletes anything.
Free trials can crawl too, paid from the trial's credits.
Keeping it current
On the schedule you picked, FormWise checks the site again, cheapest first:
- A page whose sitemap date hasn't moved is skipped without being downloaded.
- Otherwise FormWise asks the site whether the page changed since it last read it. A site that answers "not modified" costs nothing.
- Otherwise it reads the page and compares it with the version it has. Changes to dates, times and copyright years alone don't count.
Only a page that really changed is indexed again and charged. Each run checks up to 200 pages, least recently checked first, and indexes up to 50 new or changed ones, so a big site is covered over a few runs.
When pages go away
- A page that drops out of the sitemap is removed from the Knowledgebase.
- A page that answers "not found" two checks in a row is removed. Once is treated as a hiccup.
- Server errors, timeouts and pages that can't be reached are kept as they are and reported on the crawl's row. They never delete anything.
- A page the site's
robots.txtstarts blocking is kept as it is. - When a check can't list the whole site (a sitemap that couldn't be read, or a link walk that stopped early), nothing is removed that run.
Limits
- Page limit: up to 1,000 pages per crawl.
- Files per Knowledgebase: crawled pages count toward your plan's files-per-Knowledgebase limit, like any other File. See Keeping a Knowledgebase current. Pages over the limit are skipped, and the crawl says why.
- Pace: up to 50 pages are indexed per run. A bigger crawl continues in the next runs, about once an hour, until it has every page.
Managing a crawl
A crawl is listed with your other synced sources in the Knowledgebase. Its row shows its status, how many pages it holds, when it last ran and the credits that run used. From there you can change how often it checks, Sync now, or Stop syncing. Stopping leaves the pages it already read in place. Crawls from every Knowledgebase also appear on the Syncs board, next to your Drive and Notion syncs.
Starting a crawl of the same site into the same Knowledgebase again updates the existing crawl's paths, page limit and schedule instead of adding a second one.
From the API and MCP
- REST:
POST /v1/knowledgebases/{id}/crawlswithpreview: trueanswers the pages found, the room left under your plan andestimatedCredits, with nothing written. Withoutpreview, it starts the crawl (202).GET /v1/knowledgebases/{id}/crawlslists crawls with their status andlastRunCredits. Starting a crawl needs theknowledge:writescope and an org key or an owner's or admin's personal key. With no credits left, the answer is 402insufficient_credits. - MCP:
crawl_website(call it withpreview: truefirst) andlist_crawls. See MCP Server.
Include and exclude paths use the same patterns as the page picker: /docs covers everything under /docs, * matches one path segment, ** any number of segments, and a trailing $ matches that exact path only. Excludes win over includes. A start address under the root, like https://example.com/docs with no include list, crawls that section.