For a legacy website moving toward AI citations, the first technical layer is making sure AI crawlers can actually reach and read your content. That starts with two files: robots.txt and sitemap.xml.
Robots.txt: declare access explicitly
GPTBot, ClaudeBot, Bytespider and other AI crawlers check robots.txt before every crawl. Many older sites never declared AI-specific rules, so crawlers fall back to defaults and core pages can be skipped. Add an explicit User-agent block for the AI crawlers you want to allow, and keep the catch-all User-agent: * rule permissive so you do not block your own content by accident.
Sitemap.xml: help discovery
Declaring Sitemap: inside robots.txt tells crawlers where your index lives. Pages that change frequently — product pages, knowledge-base articles, release notes — deserve a higher priority. The same reasoning drives the report sitemap we publish for every [analyzed site](/sites/).
Verify before moving on
After configuring both files, test them with a crawler simulation: confirm the homepage returns 200, robots.txt allows your AI user-agents, and the sitemap parses cleanly. Only when the basics are solid do structured data and content work start to pay off.
Check your own site
Run a free [AI Readiness check](/check?url=https://example.com) to see how your robots.txt, sitemap and crawlability settings score today, and review the [API documentation](/docs/) if you want to script your own verification.