Loading tool…
Loading tool…
Loading tool…
Generate a mistake-free `robots.txt` file with presets for WordPress, Laravel, and Shopify, XML sitemap directives, and optional AI crawler rules.
Your inputs stay in your browser. Make something useful.
Loading tool…
Here’s how to get the most out of robots.txt generator.
Your robots.txt file sits at the root of your domain (/robots.txt) and instructs search engine crawlers which administrative or duplicate folders (such as /wp-admin/, /admin/, or /cart) should not be crawled, conserving crawl budget for your important landing pages.
A few things you might want to know before you start.
No! If a page is already indexed, blocking it in robots.txt prevents Googlebot from seeing a noindex tag. Use <meta name="robots" content="noindex"> to remove indexed pages.
Always upload it to the top-level public root of your domain (e.g., https://yourdomain.com/robots.txt).
Many WordPress themes and plugins rely on /wp-admin/admin-ajax.php for frontend functionality, so Googlebot needs access to render pages properly.
Run our Full Website Audit tool linked below.
