Enter any domain and we'll check if your robots.txt is blocking the AI crawlers that decide whether you show up in AI-generated answers. Free. Takes 5 seconds.
robots.txt file to explicitly allow the AI crawlers that drive visibility in ChatGPT, Claude, Perplexity, Google AI Overviews and more.AI Bot Access Checker reads your site's robots.txt and tests it against every major AI crawler we know of — the bots that fetch content to train large language models and, more importantly, the bots that fetch content live so ChatGPT, Claude, Perplexity and Google AI Overviews can cite you in answers. For each one, we show whether your site is Allowed, Partially allowed, or Blocked, and we give you a copy-paste robots.txt snippet that fixes any gaps in 30 seconds.
AI-generated answers are eating the top of the search funnel. When someone asks ChatGPT "what's the best tool for X?" or Perplexity "who makes Y?", the models can only cite sites they're allowed to read. If your robots.txt quietly blocks GPTBot or ClaudeBot, you're invisible in those answers — even if you rank #1 in classic Google search.
Most sites don't know they're blocked. Blocks usually come from default hosting rules, legacy security plugins, or an over-cautious User-agent: * line. This tool surfaces the problem in seconds, for free.
If you want to be cited in live AI answers, the most important ones to allow are OAI-SearchBot, ChatGPT-User, Claude-Web, Perplexity-User, and Google-Extended. If you also want your content to feed future model training, add GPTBot, ClaudeBot, and CCBot.
llms.txt is an emerging companion to robots.txt. Where robots.txt controls which crawlers can access your site, llms.txt tells AI models how to describe your site — a plain Markdown file at your root (https://yoursite.com/llms.txt) with a one-line summary of what you do and a curated list of your key pages. It's a positive signal, designed specifically for large language models like ChatGPT, Claude, and Perplexity. Our scanner checks whether you have one, and our free generator builds one for you in seconds. Proposed spec: llmstxt.org.
We fetch https://yoursite.com/robots.txt via a server-side proxy (so no CORS issues and nothing leaks through your browser), parse every User-agent / Disallow / Allow block using the same logic Google's robots parser uses, and resolve each AI crawler against that ruleset. We also check whether you have an llms.txt file at the root — an emerging standard that tells AI models how to describe your site in answers.
Yes. No signup, no rate limit worth worrying about, no hidden paywall. The tool is funded by an optional newsletter and occasional sponsor link — you can ignore both.
Three common causes: (1) a bare User-agent: * with a Disallow: / rule that implicitly blocks every AI crawler; (2) a security plugin (Wordfence, Cloudflare WAF, etc.) that blocks AI user-agents at the firewall level even when robots.txt permits them; (3) a CDN serving a stale robots.txt. The scan shows exactly which rule triggered the block so you can trace it.
No. AI crawlers are separate from traditional search crawlers (Googlebot, Bingbot) and the two don't interact. Allowing GPTBot has zero effect on your Google rankings — it only changes whether ChatGPT can see your content.
Training bots (GPTBot, ClaudeBot, CCBot) grab content to train future model versions — the benefit is long-term and diffuse. Answer bots (OAI-SearchBot, ChatGPT-User, Perplexity-User, Claude-Web) fetch live pages to cite in real answers — the benefit is immediate citations and referral traffic. Most sites should allow both; some content owners allow only answer bots.
llms.txt is a new standard (see llmstxt.org) — a plain-Markdown file at your site root that tells AI models what your site is about and which pages matter. It's a positive signal, in contrast to robots.txt's access-control focus. Almost nobody has one yet, which makes adding it a cheap way to stand out. Build your llms.txt in 10 seconds with our free generator →
Any time you change your robots.txt, CMS, security plugin, or hosting. Also worth a re-check every few months — AI companies add new crawler names (this list has doubled in the last year), and your rules might need updating.
Build one in seconds with our free generator. It reads your sitemap and homepage to create a ready-to-publish llms.txt template you can edit, copy, or download.