Free tool
robots.txt Generator
Your robots.txt decides which AI crawlers can read your site. Allow the answer engines you want to be recommended by, block the ones you don't, add your sitemap, and copy the result to your site root.
AI crawler access
Allowing the answer engines is what lets them read and recommend you. All allowed.
Save this as robots.txt at the root of your site (yoursite.com/robots.txt).
# robots.txt
# Generated with RankNext, ranknext.io/tools/robots-txt-generator
User-agent: *
Allow: /
# AI answer engines
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Applebot-Extended
Allow: /
User-agent: Bytespider
Allow: /
What this robots.txt generator does
This free robots.txt generator creates the small text file that lives at the root of your domain and tells crawlers what they can and cannot access. You choose which bots to allow, add your sitemap URL, and copy the result to yoursite.com/robots.txt.
Unlike a generic generator, this one knows the AI crawlers that now matter for getting recommended: GPTBot (ChatGPT), Google-Extended (Google AI), PerplexityBot, and ClaudeBot, so you can deliberately allow the answer engines you want to be cited by.
Which crawlers should you allow?
For a local business that wants to be found by AI, you generally want to allow the major AI crawlers so engines can read and recommend you, while keeping out bots that only add load. Blocking GPTBot or Google-Extended, for example, can quietly keep you out of AI answers entirely.
If you are unsure, allowing the well-known AI crawlers plus standard search engines is the safe default for visibility.
How to add robots.txt to your site
Place the file at the root of your domain so it is reachable at yoursite.com/robots.txt. Most platforms let you upload it or set it in the site settings; some frameworks generate it from config.
Once it is live, open yoursite.com/robots.txt in a browser to confirm it loads, and keep the sitemap line so crawlers can find all of your pages.
Frequently asked questions
What is a robots.txt file?
A plain text file at the root of your domain that tells web crawlers which parts of your site they may or may not access. It is one of the first things a crawler checks.
Where does robots.txt go?
At the root of your domain, reachable at yoursite.com/robots.txt. It will not work in a subfolder.
Should I block AI crawlers like GPTBot?
If you want to be recommended by ChatGPT, Perplexity, or Google AI, you generally should allow their crawlers, not block them. Blocking them can keep your business out of AI answers.
Does robots.txt affect SEO?
Yes, indirectly. It controls what search and AI crawlers can read. Accidentally disallowing important pages can hide them, while a clean robots.txt with your sitemap helps everything get found.
Want this handled for you, every month?
We build your site and get you named by Google AI and ChatGPT, proven with evidence. Start with a free check.