How Llms.txt And Robots.txt Affect AI Crawlers

Reviews Do Disproportionate Work For products more than for services, review content is the evidence base. Volume matters, recency matters more, and detail matters most, because a review that describes a specific use gives a model something to match against a specific question.

You also cannot cleanly attribute a purchase to a recommendation the buyer received three weeks earlier in a conversation you never saw. That influence is real, it is often the main value of the channel, and it will not appear in any report you own.

Where a platform lets you add structured business information alongside reviews, complete every field. These profiles are frequently cited as much for their factual details as for their ratings, and a half completed profile contributes far less than a full one even when the review count is identical. It is an hour of work per platform and it is repeatedly the cheapest improvement available.

This is the least interesting subject in the discipline and the one that most often explains a total absence from generated answers. A brand can do everything else correctly and remain invisible because a line in a text file, or a setting nobody remembers enabling, is turning the relevant crawlers away.

This entire area usually amounts to a day of work. It is routinely the difference between a brand that appears in answers and one that does not, and it is worth doing before anybody writes a single word of new content. get recommended by ai

Get the Basics Right Before Anything Clever Once access is confirmed, check that content actually exists for a crawler to read. Load your important pages with JavaScript disabled. If your specifications, pricing, service areas or contact details vanish, they are effectively absent from this channel regardless of how permissive your robots file is.

Deciding Whether to Block Anything There is a legitimate argument for restricting training crawlers, particularly for publishers whose archive is the product. That is a commercial and editorial decision and it deserves a real discussion rather than a default.

Gemini and Google Surfaces Closest to conventional search infrastructure, which has a practical consequence: work that improves your standing in Google search tends to carry over here more than it does elsewhere.

It also appears more conservative in commercial categories, hedging or declining to make a direct recommendation more often than the others. Where it does recommend, established entity signals seem to matter, which favours brands with consistent details and long records over newer entrants.

This is worth accepting rather than fighting. Your own comparison page is still worth publishing, and it will rarely be the most cited source in your category. The higher leverage move is making sure the independent comparisons that already exist describe you accurately.

Put someone's name against this. Crawler rules sit between marketing, development and whoever administers the content delivery network, which in most organisations means nobody checks them. The failures documented here are not difficult to find, they are simply nobody's job, and a quarterly review taking half an hour prevents the most complete form of invisibility available.

The last of these is the most common and the hardest to see, because it produces no error anyone internally encounters. Your site works perfectly in every browser while returning a challenge page to every legitimate retrieval agent.

Run a commercial prompt in almost any category and look at what gets cited. Review platforms, roundups and comparison sites appear first and most often, and the brands being discussed appear well down the list if at all.

This is also why review volume and recency show up so consistently in what gets cited. A platform with forty recent accounts of working with you is more informative than your own page saying customers love you, and it is treated accordingly.

The Shared Architecture All three now commonly retrieve live sources rather than answering purely from training. Your question becomes one or more searches, a set of pages is fetched and read, and the answer is composed from what was read.

Why Third Party Comparisons Dominate The comparison pages cited most often are usually not published by any of the companies being compared. A review site, a trade publication or an independent blogger weighing five options reads as disinterested in a way that a vendor's own page does not.

If the only comparisons available are written by competitors and by review sites with incomplete information about you, that is the version being used. Publishing an honest one puts a source into circulation that at least contains your figures stated correctly, and honest treatment of where you lose makes the rest of the page more credible rather than less.

Discontinued products deserve deliberate handling rather than deletion. Removing a page severs the connection between existing reviews and coverage and your catalogue, and it leaves stale third party listings pointing at nothing. Keeping the page, marking it clearly as discontinued and naming the replacement preserves the accumulated evidence and redirects the recommendation rather than losing it.