robots.txt Generator

This generator builds a standards-compliant robots.txt from multiple User-agent rule blocks. Each rule has its own User-agent, a multi-line Disallow list, and a multi-line Allow list (both edited one path per line). Global settings add an optional Sitemap URL and Crawl-delay directive, then the Generate button joins everything with correct blank-line separation between rule groups.

1. Edit each Rule card's User-agent (defaults to *, or a specific bot like Googlebot). 2. Enter Disallow paths one per line (e.g. /admin/, /api/). 3. Enter Allow paths one per line (e.g. /) for exceptions. 4. Click Add Rule to insert another User-agent block (default Googlebot); delete with the trash button when more than one rule exists. 5. Set Sitemap URL and optional Crawl-delay in Global Settings. 6. Click Generate, then Copy the output to your site root.

Output is plain text following the robots exclusion protocol: `User-agent: <agent>` followed by its `Disallow:` and `Allow:` lines, then `Crawl-delay:` if set. Rule blocks are separated by a blank line. A single `Sitemap: <url>` line is appended at the end when a sitemap is provided. Paths are emitted verbatim from your input, so leading slashes and trailing slashes must be entered as intended.