How to Block AI Crawlers and Still Show Up in Search

Do not collapse these decisions into a single score or switch. A business can make different choices for different purposes. Record who approves the policy and who puts it in place. The maintainer should not have to infer a commercial publishing decision from an ambiguous support ticket.

Google now documents a Search generative AI control in Search Console settings. It governs inclusion in specified AI search and Discover features; inclusion is the default, and child properties may inherit a parent's setting. Google states that the control does not govern AI training or other Search ranking and inclusion signals. Review the current control documentation, including its scope and inheritance behavior.

Inspect the effective value for the property you mean to change. A domain level decision and a setting on a narrower property can interact. Record the relevant property, current value, intended value, and the reason for the change. Do not assume a screenshot from a different property describes the whole site.

If the goal concerns training, follow the provider's documented control for that purpose instead. If the goal is removal from ordinary search, use the applicable indexing controls. The names of products and settings can evolve, so implementation instructions should link to current official documentation rather than rely on an old copied checklist.

OpenAI documents OAI SearchBot for ChatGPT search and GPTBot for potential training use, with independent settings. A site can allow the search bot while disallowing GPTBot. ChatGPT User supports certain user initiated actions and is not the search opt out control; robots rules may not apply to those visits. Check the official crawler documentation for current behavior and verification information.

Apply the same decision discipline to other providers: identify the product, the documented purpose of each agent, and the control that addresses your goal. A familiar looking user agent string is not enough to establish the identity or purpose of a request. Use the provider's documented verification approach when investigating logs or firewall rules.

Keep the policy readable for whoever maintains it next. A comment explaining the approved purpose helps more than a long list of unexplained bot names, which is why we keep our own file short. Review the list when a provider changes its products or documentation.

Robots.txt tells well-behaved crawlers what you'd like them to skip. It doesn't lock anything. Google's robots.txt introduction also explains its limits for preventing search appearance. Confidential records need appropriate access protection rather than a public path named in a robots rule.

Review whether a document belongs on the public website at all. Customer records, internal proposals, and private operating notes should follow the business's actual access policy. A crawler preference cannot compensate for a public file that anyone with the URL can retrieve.

This review belongs with the people responsible for website maintenance and information access. Keep it separate from marketing experiments so a visibility objective does not accidentally broaden access to material that should remain private.

The effective experience may depend on more than the website's own files. A content delivery network, firewall, hosting configuration, authentication layer, or application rule may affect access. Ask the maintainer to identify which layer produces the observed response for representative URLs.

In a hypothetical site, the robots file could allow a search crawler while the delivery layer presents it with a challenge page. In another hypothetical site, a page level indexing directive might conflict with the owner's intention to make a public resource discoverable. The diagnosis should identify the actual restriction before changing unrelated settings.

Use the indexation diagnosis process for ordinary search availability. A bot log entry alone does not prove indexing, a citation, or a ranking outcome. It is evidence of a request that needs interpretation.

Save the existing configuration and document the intended change. Identify representative pages and the expected behavior for each. Have the appropriate owner review material changes to publishing policy. Then check the effective configuration and the live response through the relevant official tools or a controlled technical inspection.

Allow for provider processing and caching rather than expecting every result to change immediately. Keep a rollback path for accidental access disruption. Record what was checked and what remains unverified, including any platform behavior that cannot be observed directly.

Connect crawler decisions to your search and AI visibility strategy. The measurement plan can help you observe relevant changes without treating a control setting as a performance guarantee.

A useful access policy states what the business wants, identifies the provider specific controls, and assigns maintenance responsibility. It preserves the distinction between being eligible for discovery, expressing a training preference, and protecting information that should remain private.