Check each public page and its access controls. Google says pages must be crawlable, return HTTP 200 and contain indexable content to meet its minimum technical requirements for Search; this does not guarantee indexing. OAI-SearchBot and PerplexityBot have separate robots.txt and firewall controls. Keep private content behind authentication because robots.txt is not a security boundary. Verify priority service pages, not only the homepage.

Updated 5 November 2026: crawler names, access controls and Google’s AI-search guidance checked against current official documentation.

Crawler access is one prerequisite for discovery, not a promise of indexing, citations or search placement. For Google Search, a page needs to be accessible to Googlebot, return HTTP 200 (a successful server response) and contain indexable content to meet Google’s minimum technical requirements. Google says meeting those requirements does not guarantee indexing. A firewall or content delivery network (CDN) can still block a request even when robots.txt allows it. Google’s technical requirements

Check the page, robots rules, rendered content and infrastructure separately.

Decide which pages should be public first

Separate public marketing content from private systems.

Public pages may include:

  • homepage;
  • service pages;
  • location or service-area pages;
  • pricing information;
  • case studies approved for publication;
  • blog articles;
  • About and contact pages;
  • public tools and documentation.

Private areas may include:

  • client portals;
  • admin screens;
  • draft invoices;
  • internal search results;
  • staging sites;
  • customer files;
  • authenticated account pages.

AI visibility is not a reason to expose private content.

Check the exact public URL first

A public service page should return its intended page without login or an access challenge. An ordinary browser or HTTP request shows what your test location can retrieve. Save the result in the access record at the end of this guide.

Use this order: choose a page that should be public, open its exact URL in a private browser window, review the applicable robots.txt rule, check Google in Search Console if you have access, and ask your site maintainer to check server and firewall logs. If you do not have Search Console or hosting access, record that and send the URL to the person who maintains the site.

A homepage that crawls successfully does not prove /services/process-safety/ or /pricing/ can be retrieved.

Use robots.txt for crawling, not privacy

robots.txt asks compliant crawlers which URLs they may request. It does not hide a URL from search by itself and does not protect sensitive information. Google says a blocked URL may still appear in results without a description if other pages link to it. Put client files, account pages and other private material behind authentication or another real access control. Google’s robots.txt guide

To find the file, open https://example.com/robots.txt with example.com replaced by your site’s domain. Read each User-agent, Allow and Disallow rule. Look for broad patterns added by a previous SEO plugin, staging setup or security hardening change. If the file does not open or a rule is unclear, ask your site maintainer before changing it.

A site-wide rule such as:

User-agent: *
Disallow: /

blocks compliant crawlers from the entire site. Look for User-agent rules naming Googlebot, OAI-SearchBot or PerplexityBot, plus rules for * that apply to other crawlers. If you are unsure what a rule does, ask your site maintainer before changing it.

A more targeted private-area rule can keep admin paths out while leaving public pages available.

Do not copy another website’s robots file. Its platform, paths and privacy requirements may be different.

Check Googlebot and Google indexing separately

Googlebot crawling is separate from Google deciding whether to index a page. For Google’s minimum technical requirements, check that Googlebot is not blocked, the page returns HTTP 200 (a successful server response), and the response contains indexable content. These conditions do not guarantee indexing. Google’s technical requirements

For an important page, confirm:

  • If your business has a verified Search Console property, paste the page URL into URL Inspection. In the indexed result, review the last crawl, indexing status and user-declared canonical. The Google-selected canonical appears only in indexed data; the live test cannot predict Google’s choice. A canonical is the page’s signal for its preferred URL, but Google can select a different one. If they differ, ask your site maintainer to check the canonical tag, redirects, internal links and sitemap. If you do not have Search Console access, ask the property owner or site maintainer to run this check.
  • Select Test live URL to check the current page fetch, crawl access and whether indexing is allowed. This checks the live page; it does not guarantee Google will index it.
  • If you need a page excluded from Google, allow Googlebot to crawl it so it can read a noindex directive, which asks Google not to show the page in search. A robots.txt block can prevent Google from seeing that directive. For private content, use access controls.
  • Keep internal links and sitemap entries consistent with the preferred public URL. A sitemap helps Google discover URLs but does not guarantee that they will be crawled or indexed. Google’s sitemap overview
  • If the page uses JavaScript, check that its main text and links are available after rendering. Google can render JavaScript, but rendering and indexing can take time; other crawlers may process JavaScript differently. Google’s JavaScript SEO guide

Google says the usual Search requirements and SEO practices apply to AI Overviews and AI Mode. It says there are no extra technical requirements or special structured data to add for those features. This does not promise inclusion in an AI feature. Google’s AI features guidance

OpenAI separates its automatic search crawler from training-related and user-triggered agents.

OAI-SearchBot is OpenAI’s crawler for surfacing websites in ChatGPT Search. OpenAI says opting out means sites will not appear in ChatGPT search answers, though they may still appear as navigational links. Allowing OAI-SearchBot does not guarantee that a page will be selected. OpenAI’s crawler documentation

Check OAI-SearchBot’s rule separately from GPTBot. If your firewall or CDN filters it, see the shared verification steps below.

Check PerplexityBot separately from user-triggered fetches

PerplexityBot is Perplexity’s automatic crawler for surfacing and linking pages in its search results. Perplexity says it is not used to crawl content for foundation-model training and recommends allowing it in robots.txt if you want your site considered for Perplexity search results. Access still does not guarantee appearance. Perplexity’s crawler guide

Treat it as a separate user agent. A rule written for Googlebot does not automatically describe what should happen to PerplexityBot. Perplexity also documents Perplexity-User, which fetches a page in response to a user’s request and generally ignores robots.txt. That user-triggered fetch is different from its automatic search crawler; do not assume a robots rule controls every user-requested fetch.

If you block PerplexityBot, record that as a deliberate access choice. A robots.txt change may take up to 24 hours to take effect, according to Perplexity.

For firewall rules and the published IP feeds for these two automatic crawlers, use the shared verification steps below.

Check that the main content reaches the page

A page can return 200 while showing little useful text until scripts run. Google renders JavaScript, but the process has separate crawling, rendering and indexing stages. Other crawlers may not handle the same scripts in the same way. For a key service page, check the delivered HTML and the rendered page, then confirm that service details and links are visible without a login or user action. Google’s JavaScript SEO guide

Keep these elements accessible:

  • page title;
  • H1;
  • service description;
  • key evidence;
  • location or service area;
  • relevant internal links;
  • contact or next-step link.

Retest after infrastructure changes

Changes to hosting and security settings can affect crawler requests even when the page text stays the same.

Retest after:

  • moving DNS;
  • enabling Cloudflare bot protection;
  • changing hosting;
  • installing a web application firewall;
  • enabling geographic restrictions;
  • adding rate limiting;
  • moving to a new JavaScript framework;
  • changing robots.txt generation;
  • adding an SEO plugin.

If the public URL returns 403, a challenge page or a redirect loop, resolve that response before treating it as a content issue.

Verify crawler requests before allowing them through

Changing a browser’s user-agent string to “Googlebot” does not prove that a request came from Google; the user-agent text can be copied. Do not add a firewall exception based on that label alone. If a Google log entry needs verification, ask your host or CDN maintainer to use Google’s current request-verification steps.

If the maintainer filters automatic search requests by IP, they should compare OAI-SearchBot requests with OpenAI’s current published range and PerplexityBot requests with Perplexity’s current published range. These feeds do not verify user-triggered agents such as ChatGPT-User or Perplexity-User; those agents have separate purposes and controls. Do not copy a saved range list into a firewall rule without checking the live feed. Keep the narrowest rule that allows the verified crawler; do not disable security protections globally.

Keep an access record for important pages

For each important public page, keep one row with the date you checked it. A successful request shows that the page was reachable from that test location at that time; it does not prove that a named crawler visited it or that a search engine will index or cite it.

The site owner can record whether the page opens without a login and copy the URL Inspection result if they have access. The site maintainer should complete the server, rendered-content, firewall and crawler-log checks. In the table, HTTP 200 means the server returned a successful response; noindex is an instruction not to show a page in search; canonical identifies the preferred URL signal. Ask the maintainer to explain any field you cannot verify.

URL and date Owner: opens without login? Search Console: crawl/index/canonical Maintainer: response/rendering/firewall/logs Action

IDJoy includes crawler and page-access checks in its AI search visibility work before recommending content changes.