LLM crawler
An LLM crawler is a bot that fetches web pages on behalf of an AI system — for training data, for live answer retrieval, or for a user's specific question. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended are LLM crawlers; blocking them makes a store invisible to the engines they feed.
In context
LLM crawlers come in three flavors: training crawlers (e.g. GPTBot, CCBot) that feed model training corpora, search/index crawlers (e.g. OAI-SearchBot, Claude-SearchBot) that build retrieval indexes for AI search, and user-triggered fetchers (e.g. ChatGPT-User, Perplexity-User) that read a page live because a user asked about it. A store’s robots.txt decides which of these get in.
Many ecommerce sites block these bots by default — often via a CDN setting nobody remembers choosing — and then wonder why they never appear in AI answers. Auditing crawler access is step one of any GEO program; it’s the first check in Veristyle’s free AI visibility audit.
Related
LLM crawler visibility for fashion catalogs covers which bots matter and how to verify access. Once crawlers are in, llms.txt and structured data determine what they understand.