Three tools, one goal: make sure AI models can read your site and describe it accurately. Here's what each tool does, when to use it, and how to get the most out of it.
AI-generated answers are replacing the top of search. When someone asks ChatGPT, Claude, Perplexity or Google's AI Overviews a question, the models decide which sources to cite based on two things: whether they're allowed to read your site, and whether they understand what your site is about.
The AI Bot Access toolkit covers both sides of that equation:
robots.txt. Our Scanner tells you which AI crawlers your current rules are blocking.llms.txt — a plain-Markdown file that tells AI models what your site does and which pages matter. Our Free Generator and AI Builder help you create one."llms" stands for Large Language Models — the AI systems behind ChatGPT, Claude, Perplexity, and Google's Gemini.
What it does: Fetches your site's robots.txt, parses it using the same logic Google's robots parser uses, and resolves each major AI crawler against the ruleset. For each bot you see Allowed, Partially allowed, or Blocked, plus an explanation of why.
robots.txt and want to verify the new rules do what you intended.robots.txt snippet that explicitly allows the AI crawlers you want.llms.txt presence as a positive signal.example.com or https://example.com)./robots.txt via a server-side proxy (no CORS issues) and analyzes it.robots.txt section and paste the result into your site's robots.txt file at the root of your domain./admin, /cart, etc. — usually fine).User-agent: * with Disallow: /, (2) a WAF/firewall rule at your CDN (Cloudflare, etc.) that blocks bots before they even see robots.txt, or (3) a stale cached robots.txt served by your CDN.What it does: Reads your site's sitemap.xml and homepage, ranks the most likely important pages, and drops them into a valid llms.txt template you can edit, copy, and publish. Purely template-based — no AI call, no cost.
llms.txt yet and want to ship one today./about, /pricing, /docs, /features, /blog, etc.# Your Site > One-line summary from your meta description ## Key pages - [Home](https://yoursite.com): replace this with a one-line description... - [About](https://yoursite.com/about): replace this...
https://yoursite.com/llms.txt.llms.txt belongs at your domain root, same place as robots.txt. In WordPress, a plugin or a rule in your theme's functions. In Next.js/Astro/static sites, drop it in the public/ folder. In Nginx/Apache, it's a file in your docroot.What it does: Crawls up to 200 of your pages, reads their actual content, and hands everything to Claude or GPT. The AI writes a full llms.txt with real, tailored, brand-aware descriptions — no templates, no placeholders. You bring your own Anthropic or OpenAI API key, so you pay only the provider's per-token cost (pennies per site, typically).
llms.txt as a deliverable for multiple clients.sk-ant-...) or OpenAI API key (sk-...) ready. Keys are entered in the tool and sent only to your chosen provider — we never store them.https://example.com).llms.txt. Edit freely, then 📋 Copy or ⬇ Download./llms.txt.llms.txt — don't let perfect be the enemy of deployed.| Goal | Best tool |
|---|---|
| Check if AI can read my site | robots.txt Scanner |
| Diagnose why a specific bot is blocked | robots.txt Scanner |
Get a quick llms.txt template for a small site | Free Generator |
Write llms.txt descriptions by hand with a good starting point | Free Generator |
Produce a polished llms.txt for a large content site | AI Builder |
Run llms.txt as an agency deliverable | AI Builder |
Generate multiple client llms.txt files fast | AI Builder |
Here's the end-to-end sequence we suggest for any site:
robots.txt and verify with a re-scan. No point writing an llms.txt if bots can't read the site.llms.txt to your domain root so it's live at /llms.txt.llms.txt is discoverable.Most often this is a bare User-agent: * + Disallow: / combination, or a security plugin/firewall that injects rules. Start by viewing your robots.txt in a browser (yoursite.com/robots.txt) and checking the raw content. If it's empty but bots still fail, suspect a CDN/WAF rule blocking by user-agent at a layer above robots.txt.
No. AI crawlers (GPTBot, ClaudeBot, etc.) are separate from traditional search crawlers (Googlebot, Bingbot). Allowing one does not affect the other.
If there's no sitemap at /sitemap.xml or /sitemap_index.xml, the generator builds a template from your homepage alone. You can still edit the result to add key URLs manually.
Reach out via the waitlist on the landing page. Codes are limited during early access.
Your key is sent only to your chosen provider (Anthropic or OpenAI). It's stored in your browser's localStorage if you tick "Remember key on this device" so you don't re-enter it. Our serverless proxy relays the one request to the provider and does not log request bodies or keys. If you prefer, untick "Remember" after each run, or click Sign out to clear everything.
The output is always shown in an editable textarea. Tighten voice, reorder sections, remove pages you don't want featured, or re-run with different instructions. The 🔄 Regenerate button reuses your domain and settings for a fresh pass.
Yes — especially the AI Builder. Nothing in the workflow identifies you or your clients to us, and the output is yours to ship under your own brand.
When you add or remove major pages, when your site's messaging changes materially, or every 6–12 months as a refresh. Unlike robots.txt, llms.txt isn't fetched on every crawl — models pull it occasionally as context, so minor page changes don't warrant re-generation.