Robots.txt Tester
Robots.txt Tester lets you read robots.txt rules, sitemap declarations, and user-agent groups. Test up to 20 paths to see which rule allows or blocks each URL.
- check_circlePublic pages only
- check_circle1 global credit
- check_circleNo setup required
Public webpages only
Run Robots.txt Tester
- 01Add source
- 02Set options
- 03Review result
Analyze public content you own or have permission to inspect. Failed and cancelled runs automatically return reserved credits.
Complete
Results
What Robots.txt Tester does
The robots.txt tester fetches a site's public robots file and separates its user-agent groups, allow and disallow rules, and sitemap declarations. You can choose a user agent and test up to 20 paths against the parsed rules.
For each path, the result shows the matching directive and whether the rule allows crawling. This makes overlapping patterns easier to debug, but it does not tell search engines to recrawl or override how a specific crawler interprets unsupported extensions.
What you can do with Robots.txt Tester
Useful fields from the start
Review user-agent groups, allow/disallow rules, sitemap declarations, path testing in one organized result.
Results you can inspect
See the returned rows, findings, and available source details before you download anything.
Export as CSV, XLSX, JSON
Keep the result in the format that fits your spreadsheet, audit, content, or development workflow.
Bounded website processing
Analyze public sources with a clear cost shown before the run and a maximum of 25 source pages.
Where Robots.txt Tester fits
Perfect for
- checkCheck a staging rule before production
- checkDebug an accidentally blocked section
- checkReview sitemap declarations in robots.txt
- checkCompare behavior for different user agents
What you get
- checkUser-agent groups
- checkAllow/disallow rules
- checkSitemap declarations
- checkPath testing
How to use Robots.txt Tester
- 01
Enter the site address
The tester resolves and fetches the public robots.txt file for that host.
- 02
Choose an agent and paths
Select a user agent and add up to 20 URL paths you want to test.
- 03
Review the matching rules
See which directive made each allow or disallow decision and update the file if needed.
What to know before you run it
- infoRobots.txt controls crawling, not guaranteed indexing.
- infoPrivate content still needs authentication and access controls.
- infoCrawler-specific extensions may not behave identically everywhere.
ExtractPics processes public content only. Use pages and files you own or have permission to inspect, and follow the source website's terms.
Questions about Robots.txt Tester
If your question is not covered here, send us a message and include the page, file, or result type you are working with.
Can robots.txt keep a page private?
How many paths can I test at once?
Does the tester change my robots.txt file?
Ready when you are
Run Robots.txt Tester
Analyze a public page to inspect user-agent groups and allow/disallow rules.
Open the tool