No robots.txt found.
No robots.txt was detected. Access rules for major AI crawlers are not declared explicitly, so they may be overlooked or left to default behavior.
- Edit the root robots.txt file and add Allow rules for the AI crawlers below.
- If you need to protect specific paths (e.g. admin panels), use precise Disallow rules inside the wildcard group.
- Add a Sitemap reference: Sitemap: https://example.com/sitemap.xml.
- Re-run the analysis after deployment to confirm the status changes to Allowed.
txt
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Bytespider
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: KimiBot
Allow: /
User-agent: *
Allow: /