Rules
User-agent
* (all bots)
Googlebot
Bingbot
GPTBot (OpenAI)
CCBot (Common Crawl)
Rules (path per line, Allow/Disallow prefix)
Add rule
Crawl-delay (seconds, optional)
Sitemap URL (optional)
robots.txt
Download