Question 1

What are bots and why do they visit websites?

Accepted Answer

Bots are automated software programs designed to perform specific tasks on the internet without human intervention. They visit websites for various purposes, including indexing content for search engines, monitoring site availability and performance, aggregating content for news feeds and platforms, generating social media link previews, and collecting data for analytics and research purposes.

Question 2

What's the difference between good bots and bad bots?

Accepted Answer

Good bots are authorized automated programs that provide value to website owners and users. They identify themselves accurately, respect robots.txt directives, and maintain reasonable request rates. Examples include search engine crawlers like Googlebot and Bingbot. Bad bots are designed to exploit websites for malicious purposes, often disguising their identity and engaging in harmful activities such as content scraping, credential stuffing, and DDoS attacks.

Question 3

Why should I whitelist bots instead of blocking all of them?

Accepted Answer

Blocking all bots would severely harm your website's performance and visibility. Search engine crawlers need access to index your content—without them, your pages won't appear in search results. Social media bots enable proper link previews, monitoring bots track uptime, and content aggregators distribute your content to wider audiences. Whitelisting allows you to selectively permit beneficial bots while maintaining security against malicious ones.

Question 4

How do I whitelist bots in Cloudflare?

Accepted Answer

Cloudflare provides multiple methods for whitelisting bots. The most effective approach uses Custom Rules in the Web Application Firewall (WAF). Navigate to Security → WAF → Custom Rules, then create a rule using the cf.client.bot field to identify verified bots. For custom bots not in Cloudflare's verified directory, use IP Access Rules by going to Security → WAF → Tools → IP Access Rules.

Question 5

Which bots should I whitelist?

Accepted Answer

Your whitelist should include all major search engine crawlers such as Googlebot, Bingbot, Slurp, DuckDuckBot, Baiduspider, and Yandex Bot. Social media platform bots including Facebook's crawler, Twitter bot, LinkedIn bot, and Pinterest bot are also essential. Additionally, whitelist monitoring and analytics services you actively use, such as Pingdom, UptimeRobot, and StatusCake.

Question 6

What happens if I accidentally block good bots?

Accepted Answer

Blocking legitimate search engine crawlers leads to immediate and severe consequences. Your pages will gradually disappear from search results, new content won't be indexed, and your search rankings will decline, potentially resulting in 50-90% drops in organic traffic within weeks. Social media sharing will break, monitoring services will report false alerts, and content aggregators will stop featuring your articles.

Frequently Asked Questions

Want to learn more?