AI Crawler Robots.txt Generator – Block or Allow AI Bots

Generator Settings

Quick preset

Blocks model-training and answer-engine bots while leaving Googlebot, Bingbot and other classic search crawlers untouched.

Site URL (for the sitemap line)

AI crawlers

Checked = blocked (Disallow: /). Unchecked = allowed.

Search & social crawlers

These drive organic traffic and rich previews. Blocking them removes you from search results.

Additionally disallow these paths for every bot

One path per line. * and $ wildcards are supported by Google and Bing.

Extras

Include Sitemap lineAbsolute URL, as the spec requires. Add Crawl-delayHonoured by Bing and Yandex; ignored by Google. Include explanatory commentsNames each bot's operator so future-you knows why. Group blocked bots into one ruleShorter file; one User-agent list sharing a Disallow.

Generated File ready

robots.txt0 lines