Claude may surface a business when its search or user-directed fetch can access relevant public pages, but allowing Anthropic's bots does not guarantee a mention or recommendation. Check Claude-SearchBot, which indexes pages for search, and Claude-User, which can fetch pages for a user query. ClaudeBot is for model training and is a separate choice.
Updated 4 November 2026: Updated Anthropic’s crawler guidance, separated search and user-directed fetches, and clarified what available usage data can show.
Which Anthropic bots affect whether Claude can find your pages?
Anthropic documents three website bots. They serve different purposes, so decide on each one separately:
- Claude-SearchBot discovers and indexes pages to improve Claude’s search results. Blocking it can prevent your pages from being indexed for search.
- Claude-User retrieves pages when a person asks Claude a question. Blocking it can reduce access for user-directed searches.
- ClaudeBot collects public web content that may contribute to model training. It is separate from search and user-directed retrieval.
Anthropic says these bots respect robots.txt, a plain-text file at the root of a site that gives crawlers access instructions. Set the rules separately for each host or subdomain you want to control. If you want your pages available to Claude Search, check that the rules do not block Claude-SearchBot or Claude-User. That makes pages accessible to those routes; it does not ensure a mention, citation or recommendation.
Anthropic’s help centre links to a current list of IP (internet protocol) addresses, the network addresses a site may record with each request. It warns that blocking those addresses may not reliably establish an opt-out and can prevent the bots from reading robots.txt. When checking requests, ask your host to compare the bot name and recorded address with Anthropic’s published list. Do not treat an IP block as a substitute for the documented robots.txt controls. Read Anthropic’s crawler guidance.
Is Claude Search the same as a direct page fetch?
No. Claude’s web search can return current web results and source links. When web search is available and enabled, Claude can also fetch a supplied page URL directly. A direct fetch checks the page the person named; it does not show whether Claude would discover it for a broader search query. Claude’s Help Center explains web search and web fetch.
When Claude uses web search, it provides citations and source links. Open them to see what supports the response; a citation does not prove its claims or explain why Claude named a business. Read Claude’s web-search guidance.
Claude in Chrome is a browser assistant operating through a person’s Chrome session. It is a separate kind of use from the three named website bots. A browser session is not evidence that Claude-SearchBot indexed a page.
How can I check whether Claude returns your business?
Use queries a real customer might ask, then record what Claude returns. Do not treat one response as a site-wide score.
Try a small set such as:
- “Tell me about [business name] and link to the pages you used.”
- “Which [service] providers serve [city]?”
- “Who can help a [customer type] with [specific problem] in [area]?”
- “Compare [business A] and [business B] for [specific need]. Cite their published service details.”
For each run, note the date, exact prompt, country or location, Claude product or mode, whether web search was indicated, and the cited URLs. Record whether Claude named the business, got its service and location right, and linked a page that supports the claim. If search is unavailable in the account or workspace, record that instead of treating a no-mention response as a search result.
Repeat the same prompts under similar conditions after you change an important business fact, service page or access rule. Use a new conversation for a separate prompt. The comparison can show what changed in your own checks; it cannot reveal Claude’s internal ranking factors.
What can public usage data tell a South African business?
EZY Research reviewed server requests from 125 connected business websites and reported that Claude accounted for 3.2% of AI fetches triggered by user activity in its April-to-July 2026 sample. Its sample skews toward small businesses whose owners sought an AI-visibility tool, so it is not an estimate of Claude’s share of South African customers or business enquiries. Read the study and methodology.
Use your own customer questions, site referrals and qualified enquiries to decide how much time to spend checking Claude. Referral data can miss Claude-assisted visits when an app strips referrer details, so pair analytics with a simple “How did you find us?” question. South Africa appears on Anthropic’s supported-regions list, but regional availability alone does not tell you whether a particular account can use web search. Check Anthropic’s supported regions.
What should I check if Claude gives a wrong answer or omits the business?
First confirm Claude used web search and note the prompt and location. Then check the source pages it did cite, if any, and compare them with the business’s current service, location and contact details.
If a page appears inaccessible, ask your developer or hosting provider: “Does our site’s robots.txt allow Claude-SearchBot and Claude-User, and do the server or security-service logs show requests from an address on Anthropic’s published crawler list?” Check that the relevant page loads for an ordinary visitor and that a firewall or bot challenge is not blocking it.
Do not add unsupported credentials, reviews or claims to influence a response. Correct outdated business facts on the pages and public profiles you control, then repeat the same queries and record the sources.
Joy Mahlatjie is IDJoy’s founder and the author of this guide. IDJoy offers AI search visibility services. Review the service scope and pricing before deciding whether outside help would be useful.