AI crawler directory · Google
Googlebot
Powers Google Search including AI Overviews and AI Mode, and feeds the web-crawl side of the Shopping Graph behind Gemini shopping.
The verdict
Blocking Googlebot costs real visibility — it is the crawler that feeds the surface, not a training bot.
18% of the 50 top DTC stores in Shelfgrade's weekly readiness panel currently block Googlebot (2026-W36).
robots.txt
Recommended: allow
User-agent: Googlebot Allow: /
Costs visibility
User-agent: Googlebot Disallow: /
Identifying it in your logs
Published user-agent string
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Version segments drift — match on the Googlebot token, not the full string.
Questions this page answers
What is Googlebot?
Googlebot is Google's crawler token. Powers Google Search including AI Overviews and AI Mode, and feeds the web-crawl side of the Shopping Graph behind Gemini shopping. Source: the vendor's own crawler documentation (https://developers.google.com/search/docs/appearance/ai-features), last re-verified 2026-08-27.
Does blocking Googlebot in robots.txt hurt my store's AI visibility?
Blocking Googlebot costs real visibility — it is the crawler that feeds the surface, not a training bot.
What does the Googlebot user agent look like in server logs?
The vendor-published UA string is: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) — note that version segments drift over time; match on the "Googlebot" token, not the full string.
Primary source: https://developers.google.com/search/docs/appearance/ai-features — every Shelfgrade rule links the vendor doc it derives from, re-verified weekly.
Other Google crawlers: Google-Extended · GoogleOther