Which AI crawlers should an RIA allow for search, and which controls cover model training?

An RIA should separate AI-search access, model-training controls and user-requested page retrieval before changing crawler rules. They are different uses, and the vendors do not give them one shared switch. Start with the firm's goals and the vendor's current documentation. Then have the website owner review the exact public pages, crawler rules and security controls. Allowing a crawler is not a promise that an answer engine will recommend the firm.

This guide compares public vendor controls checked on October 2, 2026. It is an educational review method, not a recommendation to allow or block a bot on Valora's site or your firm's site. It does not change any website policy.

What decision is the firm actually making?

The useful question is not "Should we block AI?" It is "Which public content may be used for which purpose?" That wording exposes decisions hidden inside a broad label.

A firm might want public educational pages to be discoverable in search while taking a different position on training use. It might also have client-only documents that should require authentication regardless of the crawler. Those are separate decisions. Do not let a generic bot-blocking setting decide all of them by accident.

Write down the public content in scope, the use under review and the person who owns the decision. Ask the technical reviewer what the current rules do before asking for a change. A policy preference and a verified server response are different kinds of evidence.

The worksheet below is for that review. Its decision column deliberately stays open. It does not supply a recommended rule, a legal conclusion or a ready-to-paste site configuration.

How does OpenAI separate search from training?

OpenAI names separate controls for automatic search crawling and content that may be used for model training. Do not treat the word "GPT" as a complete description of a crawler's purpose.

In its Overview of OpenAI Crawlers, checked on October 2, 2026, OpenAI describes OAI-SearchBot as a search crawler for ChatGPT's search features. It describes GPTBot as a crawler for content that may be used in training its generative AI foundation models. The documentation says the settings are independent: a publisher can allow OAI-SearchBot while disallowing GPTBot.

That is a vendor-specific distinction, not a claim that every AI provider uses the same arrangement. OpenAI also says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Neither allowing access nor seeing a navigational link proves that the firm has earned a recommendation.

The same page publishes crawler details and IP lists. A technical reviewer should use those current materials rather than a copied user-agent example from an old blog post. Keep the documentation date and the site's observed behavior in the record.

Where does a user asking an AI tool to read a page fit?

User-requested retrieval is different from automatic crawling. Review it as its own category, with the behavior described by the specific vendor.

OpenAI's crawler page says ChatGPT-User supports certain actions initiated by users. It is not used to crawl the web automatically or determine whether content can appear in Search. The documentation warns that robots.txt rules may not apply to these user-initiated actions. Do not tell a client that one crawler rule blocks every way a page can be retrieved.

Anthropic describes a different control in its web-crawling guidance, checked on October 2, 2026. Claude-User supports retrieval when a person asks Claude a question. Anthropic says disabling Claude-User prevents its system from retrieving the site's content in response to a user query. Its page also says its bots honor robots.txt signals.

These descriptions should not be flattened into "user bots ignore robots.txt" or "all user bots obey it." Put each vendor's stated behavior beside the correct bot name. If the vendor changes that statement, the firm's review needs to change too.

How does Anthropic separate its three bots?

Anthropic's documentation separates ClaudeBot, Claude-SearchBot and Claude-User. Read the purpose attached to each name before interpreting a block.

The guidance describes ClaudeBot as collecting web content that could contribute to model training. Restricting it signals that future materials should be excluded from Anthropic's training datasets. That wording concerns future collection; it is not evidence that an earlier training dataset has been edited or that all previous use has been undone.

Claude-SearchBot is described as improving search results. Anthropic says disabling it prevents its system from indexing the site's content for search optimization and may reduce visibility and accuracy in search results. Claude-User, discussed above, concerns user-directed access. Keep all three rows separate.

The page also explains that a robots.txt change must cover each relevant subdomain. A rule on one host should not be assumed to apply to another. This matters when an advisory firm's public articles, main website and client portal use different hosts.

Is Google-Extended another search crawler?

No. Google's current documentation describes Google-Extended as a control token, not a separate HTTP request user agent. Its scope also goes beyond a simple training-only label.

In Google's common crawlers documentation, checked on October 2, 2026, Google says Google-Extended controls whether content Google crawls may be used for training future Gemini models and for the specified Gemini and Vertex AI grounding uses. The same documentation says Google-Extended does not affect inclusion in Google Search and is not a Google Search ranking signal.

Do not turn that statement into a blanket claim about every Google AI feature. Keep the named products and uses in the review. "Grounding" and "training" are different uses even when one token covers both.

Google says crawling for this control uses existing Google user-agent strings. A log entry is therefore not expected to carry a distinct Google-Extended HTTP identity. The reviewer needs the documented control behavior as well as the site's rules, not a search for a nonexistent separate crawler name in the logs.

What should the review worksheet contain?

Use a row for each use and control. Record what the vendor says, what the firm wants and what the technical reviewer has actually checked.

This is an original organizing worksheet, not a vendor-prescribed form. It intentionally contains no allow or disallow recommendation. Fill the decision field only after the owner and qualified reviewers have settled the policy for the actual site.

Why is robots.txt not enough for client information?

A crawl preference is not a password. Private information needs real access controls, not merely a request that a bot stay away.

Google's robots.txt introduction, checked on October 2, 2026, says not all crawlers support its rules. It also explains that a disallowed URL can still appear in search when linked elsewhere. Its practical rules guide warns that the file itself is public and can disclose private paths.

Have the security owner review authentication and authorization for client-only files. A publicly readable PDF does not become confidential because it is listed in robots.txt. Do not test with real client records or expose a sensitive path in a public checklist.

Crawl controls and indexing controls are different too. Google's documentation describes robots.txt primarily as a crawling control and identifies other methods for keeping a page out of Search. Changes to private pages, noindex rules or sitewide access require their own review.

How should the firm check a proposed change?

Give the technical reviewer a narrow question and preserve the result. "AI is blocked" is too broad to verify.

The record should identify the exact host, public URL, bot or product token, current rule and intended use. The reviewer can then inspect the relevant response and server configuration. Google's robots.txt specification, checked on October 2, 2026, describes file location, scope and rule matching. Use the relevant vendor's documentation rather than assuming every crawler interprets every extension identically.

Keep a dated before-and-after record if a change is approved. Record uncertainty when logs or firewall settings cannot establish the answer. Do not report an owner preference as a completed technical change or a successful fetch as a confirmed recommendation.

How is this different from an indexability or recommendation check?

This review answers which uses a public site's controls address. It does not establish that a page is indexed, useful enough to cite or likely to win a client.

Use the existing RIA indexability guide for page access, noindex, canonical and Search Console questions. Use the ChatGPT and Perplexity recommendation guide for the broader content and evidence discussion. Neither guide should be read as a guaranteed ranking or recommendation service.

When you report results, keep the stages separate: an intended policy, a verified configuration, a successful fetch, an observed search appearance and an actual inquiry. A bot allowed through the door is still not a client walking through it.