What one line in a Shopify review taught us about the bots nobody optimizes for

We read a lot of our own Shopify App Store reviews, mostly because founders like knowing what customers actually say rather than what we assume they’d say. Most of them mention visibility, or traffic, or “finally showing up in ChatGPT.” One from a streetwear brand based in the US said something shorter and easy to skim past: “The app doesn’t just handle product listings and styling, it’s built with AI and LLM crawler technology in mind.”

We kept coming back to that line, because “crawler technology” isn’t the part of this conversation most brands are thinking about yet. Everyone we talk to has heard of AI search optimization by now. Almost nobody has asked us whether their site is actually reachable by the bots doing the reading.

Googlebot isn’t the one you need to worry about anymore

For twenty years, “crawler” meant Googlebot, maybe Bingbot, and your job was sitemaps and page speed. That’s not the full list anymore. ChatGPT uses GPTBot and OAI SearchBot. Perplexity runs PerplexityBot. Anthropic runs ClaudeBot. Google itself has a separate crawler, Google Extended, that feeds its AI Overviews and behaves differently from classic Googlebot. These are distinct pieces of software, with different rules, different rendering behavior, and different tolerance for JavaScript.

That last part matters more than it sounds like it should. A number of these AI crawlers don’t execute JavaScript the way a browser does. If your product data loads client side, meaning a shopper sees it fine but the page’s raw HTML is mostly empty until scripts run, several of these bots see nothing. Not a bad description. Nothing.

The quiet ways a store blocks its own visibility

None of this usually happens on purpose. In the audits we run, the same handful of issues show up again and again:

• A security or bot-management app blocks “unknown” user agents by default, and GPTBot or ClaudeBot gets caught in that net along with the scrapers it was meant to stop.
• robots.txt was copied from a template years ago and disallows a list of bots nobody had heard of at the time, which now happens to include half the AI ecosystem.
• Product data renders through JavaScript with nothing meaningful in the initial HTML, so a bot that doesn’t run scripts gets a blank shell.
• There’s no llms.txt or structured feed at all, so even a crawler that reaches the page has no clean, machine readable version of what’s actually for sale.

Any one of these quietly removes a brand from consideration before an AI assistant ever gets to the part where it decides whether the product actually matches what the shopper wants. It doesn’t matter how well written your product copy is if the thing reading it can’t see it.

What “built with crawler technology in mind” actually looks like

This is the part that review was pointing at, even in one short sentence. Being ready for AI crawlers comes down to a few concrete things: your product and content pages need to be crawlable without a blanket noindex or an overzealous bot blocker, your important content should live in the server rendered HTML rather than behind client side JavaScript, and you need a clean, structured feed, an llms.txt or equivalent, that gives these bots something explicit to read instead of asking them to guess.

That’s the layer Veristyle builds underneath a storefront. We generate the structured feeds these crawlers actually consume, we check whether a site is accidentally locking them out, and we keep that feed current as a catalog changes, so a model asked “which brands make relaxed linen shorts for warm weather” has something real to match against instead of a wall it can’t get past.

Why this is the unglamorous half of AI visibility

Semantic tagging and rich product attributes get most of the attention in this space, and for good reason, since that’s what determines whether you show up in the right search once a crawler reaches you. But none of that matters if the crawler never gets there in the first place. It’s the difference between writing a great answer and making sure the question ever reaches you. We’d rather brands fix the boring, structural half of this first, because it’s usually a fast check, and because it’s the kind of gap you don’t notice until a competitor starts showing up in searches you should have won.

If you want to know whether your own site is actually reachable by GPTBot, ClaudeBot, and the rest, an AI visibility audit checks this in about five minutes, no CMS access required. Or if you’d rather walk through what this looks like for your specific catalog, book a demo and we’ll show you exactly what’s blocking you today.