Reference · re-verified weekly
The AI crawler directory
Every crawler that matters for AI shopping visibility, from the vendors' own documentation — what each bot actually feeds, and the exact cost of blocking it. Most “block the AI bots” advice conflates training crawlers (safe to block) with the search crawlers that gate whether your store appears at all.
Blocking these costs visibility
OAI-SearchBotOpenAIGates ChatGPT search and shopping answers — opted-out sites are not shown in ChatGPT search answers (navigational links only)
PerplexityBotPerplexityIndexes pages for Perplexity search and shopping results — Perplexity recommends allowing it for your site to appear
GooglebotGooglePowers Google Search including AI Overviews and AI Mode, and feeds the web-crawl side of the Shopping Graph behind Gemini shopping
Narrow or no visibility effect
OAI-AdsBotOpenAIValidates pages submitted as ChatGPT ads and informs ad relevance; only visits pages submitted as ads
GPTBotOpenAICrawls content for OpenAI model training only — blocking it does not affect ChatGPT search or shopping visibility
ChatGPT-UserOpenAImay ignore robots.txtFetches pages on behalf of a ChatGPT user; robots.txt rules may not apply and it does not affect search inclusion
Perplexity-UserPerplexitymay ignore robots.txtFetches pages when a Perplexity user asks a question; documented as generally ignoring robots.txt
Google-ExtendedGoogleControls Gemini model training and grounding only — explicitly does not impact Google Search inclusion, ranking, or Shopping
GoogleOtherGoogleGeneric Google crawler used by various product teams; blocking it affects no specific product
Want to know which of these your robots.txt blocks today? Run the free 30-second scan — or start with the AEO Quickstart.