Bots
A bot is a software application programmed to do certain tasks.
Bots can be used for good (chatbots, search engine crawlers) or for evil (inventory hoarding, credential stuffing).
Cloudflare maintains an internal list of verified bots that are associated with search engine optimization (SEO), website monitoring, and more.
You can use this list to prevent any bot protection measures from impacting otherwise helpful bots, such as search crawlers.
For a partial list of verified bots, refer to Cloudflare Radar ↗.
You can block artificial intelligence (AI) bots, crawlers, and scrapers from scraping your website content and training large language models (LLM) to recreate it without your permission.
When you enable this feature via a pre-configured managed rule, Cloudflare can detect and block verified AI bots that comply with robots.txt and respect crawl rates, and do not hide their behavior from your website. The rule has also been expanded to include more signatures of AI bots that do not follow the rules.
The rule to block AI bots takes precedence over all other Super Bot Fight Mode rules. For example, if you have enabled Block AI bots and Allow verified bots, verified AI bots will also be blocked even if you allow other verified bots on your website or application.
For Bot Management customers, if you have set a rule to serve managed challenges to definitely automated bots, AI bots will also be challenged because custom rules run in a phase before Super Bot Fight Mode, which is the phase when the rule to block AI bots runs.
This behavior remains the same if the setting for verified, definitely automated, and likely bots is set to block or allow. If you have an action to allow for these rules, the request is not matched to any rule and proceeds to the next ruleset phase. Similarly, if the action is set to block, they will be blocked in the earlier phase and do not move on to match the AI rule at all. However, when the action is challenge, the request matches a rule and therefore will not be matched to any rules after.
For self-serve non-Bot Management customers, all rules for verified, definitely automated, and likely bots run in the phase following the AI bots rule.
--- title: Rule phases --- flowchart LR accTitle: AI bots rule phases diagram accDescr: This diagram details the phases in which AI bots rules run. A[Custom rules] --> B[Block AI bots<br>managed rule] --> C[Allow verified bots rule]
This feature is available on all Cloudflare plans.