Robots.txt Builder

Build a clean robots.txt file for launch checklists, migrations, and crawl control.

Site Setup

Site URL

Used to generate the sitemap line. Keep the protocol included.

Sitemap URL Primary user-agent

Start broad with `*` unless you need bot-specific rules.

Block admin and private areasAdds common disallow rules such as `/admin/`, `/cart/`, `/checkout/`, and `/search`. Block parameter-based crawl trapsAdds common parameter patterns for sort, filter, and session URLs when appropriate. Add AI crawler noteCreates an extra comment block so you can review how you want to handle newer AI crawlers.

Custom allow rules

One path per line. Use only paths, not full URLs.

Custom disallow rules

One path per line. Useful for staging, temporary folders, or thin utility paths.

Quick reminder Robots.txt controls crawling, not indexing. If a URL is already known, you may still need `noindex`, canonical tags, redirects, or authentication depending on the case.

Generated File

Preview Summary

User-agentAll bots (*)

Sitemaphttps://learnseohub.com/sitemap.xml

Core blocks/admin/, /cart/, /checkout/, /search

Custom allow rulesNone yet

Custom disallow rulesNone yet

Best use caseUse this before launch, after a redesign, or any time you need to formalize crawl boundaries.