robots.txt Crawling Guide
What does a robots.txt tester check?
It fetches the public robots.txt file and evaluates whether major search engines and AI crawlers are allowed or blocked by crawl directives.
Can robots.txt protect private content?
No. robots.txt is a crawl instruction for compliant bots, not an authentication or firewall control. Private content must require access control.
Should AI crawlers be allowed?
Allowing AI crawlers can improve discovery in AI search products, but publishers may block selected bots depending on licensing and content policy.