Robots.txt generator with sitemap
Build a robots.txt file with user-agent rules and your sitemap locations.
Add your input
Runs in your browser. Nothing is uploaded.
Your result
Your result appears here.
What does Robots.txt generator with sitemap do?
Write a robots.txt file with user-agent groups, allow and disallow rules and the location of your sitemaps.
How to use this tool
- 01
Add a user agent and the paths it should not crawl, plus any paths it may.
- 02
Add the full URL of each sitemap, then choose Generate robots.txt.
- 03
Review the warnings, download robots.txt and place it at the root of your site.
Example inputs
Use this example as a starting point, then replace it with your own details.
Limits and what to check
It runs in your browser and nothing is uploaded. Line breaks and comment characters are refused in values so a rule cannot inject another directive. robots.txt is a request to compliant crawlers, not access control, and it does not guarantee indexing or ranking.
This tool runs in your browser and does not use an AI model.
Frequently asked questions
Using the tool
Q1Does robots.txt hide or protect pages?
No. It is a request that well-behaved crawlers follow. It is not access control, and a blocked URL can still appear in search results if other pages link to it. Use authentication or a noindex tag instead.
Q2Why am I warned about Disallow: /?
That rule asks every compliant crawler to stay away from your whole site. It is rarely what a public website wants, so check it before you publish.
Q3Is my robots.txt created on a server?
No. It is written in your browser and nothing is sent to AI Smarty. Copy or download the file when you are happy with it.
Q4Will a robots.txt file improve my rankings?
No. It only tells compliant crawlers which paths to avoid and where your sitemaps are. Search engines still decide what to index and rank.
AI Smarty for your website
Let customers ask.
Let your AI agent answer.
Create an AI agent using your website and documents. Give visitors a way to find answers and leave an enquiry, even when your team is unavailable.
Starter and Growth include a 7-day trial.
Do you offer weekend appointments?
An illustration of an agent on your website.
Creating an agent is separate from using this tool. Your tool inputs are not added to an agent automatically.