Robots.txt

Seventy-five percent of websites that actively block AI crawlers through robots.txt, meta tags, or server-level rules still appear in AI-generated answers from ChatGPT, Perplexity, and Gemini. Blocking does not stop citations. It stops you from controlling them. That number comes from new cross-platform citation analysis published by Position Digital in April 2026, and it dismantles the most common instinct brands have when they discover AI engines are using their content: shut the door. ...

llms.txt is the standard that tells AI engines what your website is about, what content is available, and where to find it. Wikipedia updated its SEO article on April 19, 2026 to include generative engine optimization (GEO) as the prevailing term for LLM-based search optimization. Yet 95% of websites still do not have an llms.txt file. This is the robots.txt moment for AI discovery. The brands that implement llms.txt now will control how AI engines like ChatGPT, Perplexity, and Gemini understand and cite their content. The brands that wait will be invisible. ...