- User-agent rules
- Disallow directives
- Allow exceptions
- Sitemap declarations
- Unexpected global blocks
40 LIVE TOOLS
Outils Web gratuits
Des contrôles rapides et 24 diagnostics fonctionnels pour le SEO, la sécurité, la performance et le développement.
SEO
40 toolsrobots.txt Tester
Fetch robots.txt and summarize crawler directives.
NO WEBSITE SELECTED
Start Website Scan →
Start with a website scan first.
Choose a website from the RUTSS homepage. After that, every tool here will analyze the same website automatically.
ABOUT THIS CHECK
robots.txt Tester
Fetches the live robots.txt file from the site root.
WHY IT MATTERS
robots.txt controls crawler access patterns and can accidentally block important sections.
Important public pages are crawlable while private/admin paths are intentionally restricted.