- User-agent rules
- Disallow directives
- Allow exceptions
- Sitemap declarations
- Unexpected global blocks
40 LIVE TOOLS
Free Website Tools
Fast checks. Zero clutter. A growing suite of working diagnostics for SEO, security, performance and development.
SEO
40 toolsrobots.txt Tester
Fetch robots.txt and summarize crawler directives.
NO WEBSITE SELECTED
Start Website Scan →
Start with a website scan first.
Choose a website from the RUTSS homepage. After that, every tool here will analyze the same website automatically.
ABOUT THIS CHECK
robots.txt Tester
Fetches the live robots.txt file from the site root.
WHY IT MATTERS
robots.txt controls crawler access patterns and can accidentally block important sections.
Important public pages are crawlable while private/admin paths are intentionally restricted.