Website monitoring
Website monitoring groups pages in knowledge-base folders. Discovery finds new sources; content checks look for changes to pages already included. These have separate automatic schedules, and you can also request them manually.
Recurring automatic checks require a paid plan. Manual actions are described below.
1. How to enable monitoring
1.1 When adding a new source
- In the knowledge base, click Add resources and enter a website or section URL on Web.
- Open Advanced settings and choose a Look for new sources frequency instead of the default Never.
- This enables hierarchy mirroring and locks its switch. Enable Automatically include new sources if you do not want to approve new results manually. It is off by default.
- Choose a discovery method and, optionally, depth, limit, and include/exclude paths. The tracked website saves these settings for future scans.
- Click Search, select sources, and confirm Add.
You can enable Mirror website hierarchy without scheduled discovery. Organizing sources into folders alone does not enable automatic discovery. Direct documents and This URL only do not use scheduled discovery of new sources.
See Adding sources for methods and regex examples. Listing discovery includes visited listing pages as well as their documents; deselect unwanted content before import.
1.2 Retroactively on an existing source
- In the left menu, open Knowledge base.

- Select the desired source in the list and in the three-dot menu above the list click Monitor entire website.

- In the dialog, select the website domain — under this name a folder will be created in the list that groups all pages of that website. The number in parentheses is the count of sources with this domain that you selected.

- Optionally check Also move all existing website documents — other URLs from the same website that you already have in the knowledge base will be moved into the same folder.

- Confirm.
2. Page statuses
Each page of a monitored website has a status that indicates how Kanbu handles it.
New
The page was newly discovered during the last crawl. It's not yet in the knowledge base — you can add it or ignore it.
Existing
The page is included in the knowledge base. Its content checks follow the automatic-update settings; you can also request a check with Crawl and refresh. Discovery alone does not refresh its stored content. However, Follow links and scrape reads content during discovery and incurs usage for that work.
Removed
A source check confirmed that the page is unavailable. Absence from discovered URLs alone is not evidence of removal. What happens to it depends on the Behavior on removal setting.
Not monitored
You manually excluded the page from monitoring. Kanbu doesn't check or update it, but it remains in the list — you can include it again anytime.
3. Behavior on removal
Determines how confirmed unavailable sources are handled. You'll find the setting in the knowledge base after selecting a monitored website and clicking the Edit button above the list.

- Flag — the page stays in the knowledge base and switches to Removed status. The decision what to do next is up to you. Recommended option for most cases.
- Remove — the page is automatically removed from the knowledge base.
- Ignore — the page stays in the knowledge base unchanged, even if it's missing from the website.
Incomplete discovery, reaching a limit, or an empty result does not by itself remove stored sources.
4. Manual crawl and refresh
Open the monitored website in the knowledge base and use the Crawl dropdown above its contents:
| Action | What it does |
|---|---|
| Crawl | Discovers sources using the saved method and rules. New sources follow your automatic-inclusion setting. Absence of a URL alone does not trigger removal. |
| Crawl and refresh | Also checks the content of included pages across the website's subfolders. Changed content is refreshed when it exceeds the configured change threshold. |
Both actions run in the background. You can leave the page; changes may take up to 20 minutes to appear. Large websites, retries or service delays can take longer. A queued crawl does not mean the refreshed content is ready for answers yet.
Crawl is usually cheaper, but automatically importing new pages still uses processing credits. Crawl and refresh can use more credits for checking and processing content. Unchanged pages avoid unnecessary reprocessing where change detection can establish that they are unchanged.
Manual actions work when the automatic schedule is Never, organization-wide automatic updates are disabled, or your plan does not include automatic updates. They do not enable automatic updates. Usage limits still apply; requests wait when the usage limit is exhausted and no credit remains. Use Sync now for Google Drive and RSS sources.
The Content updated column shows when saved document content last changed, including its initial import. An unchanged check does not change this date. Hover over the date for Last checked. Folders, unimported pages and older content without a known update date show —.
Frequently asked questions
-
I add monitoring retroactively — what happens to pages I already have in the knowledge base?
If you check the option to move existing documents when configuring, Kanbu will place them under the selected domain and connect them to the monitoring structure. Their content is not lost or reprocessed.
-
I enabled automatic inclusion of new pages — will all of them really be added?
Eligible new results are automatically included according to saved rules and available limits. Restrict unwanted sections with Exclude paths (regex), or disable automatic inclusion and select results manually. Partial discovery may not find every source.
-
How do I know the crawl completed?
Check Last scanned above the list and any source error message. Newly discovered pages appear as New or are automatically included, depending on your settings. Absence from results alone does not mark a source as removed. A completed crawl does not mean content checking and processing have finished.
Related
- Updating sources — automatic refresh of existing page content
- Adding sources — how to add a source and configure monitoring when adding
- Bulk actions — bulk operations on monitored website pages