Check crawler access at the page and infrastructure level. Confirm important public URLs return successful status codes, robots.txt does not unintentionally block Googlebot, OAI-SearchBot or PerplexityBot, noindex and canonical tags are correct, the CDN is not challenging legitimate crawlers and core content is available without login. Keep private portals blocked. Test the exact service pages you want discovered, not only the homepage.

A page cannot be cited from the public web if the system cannot retrieve it.

That sounds obvious, but crawler access often breaks outside the CMS. A robots.txt file can be correct while a CDN blocks the request. A page can return 200 OK while a noindex directive tells Google not to index it. A JavaScript shell can load while the useful service information never appears to a crawler.

Check the whole route.

Decide which pages should be public first

Do not begin by allowing every bot everywhere.

Separate public marketing content from private systems.

Public pages may include:

  • homepage;
  • service pages;
  • location or service-area pages;
  • pricing information;
  • case studies approved for publication;
  • blog articles;
  • About and contact pages;
  • public tools and documentation.

Private areas may include:

  • client portals;
  • admin screens;
  • draft invoices;
  • internal search results;
  • staging sites;
  • customer files;
  • authenticated account pages.

AI visibility is not a reason to expose private content.

Check the exact URL with a normal request

Start with the page you want discovered.

A public service page should usually return a successful response without requiring cookies, login, CAPTCHA or a location-specific exception.

Record:

Check Result
Final URL
HTTP status
Redirects
Canonical URL
Robots meta
Login required
Main content visible

Do this for the service pages, not only /.

A homepage that crawls successfully does not prove /services/process-safety/ or /pricing/ can be retrieved.

Review robots.txt intentionally

robots.txt controls crawler access patterns. It is not an access-control system for sensitive data.

Read every User-agent, Allow and Disallow rule. Look for broad patterns added by a previous SEO plugin, staging setup or security hardening change.

A rule such as:

User-agent: *
Disallow: /

blocks compliant crawlers from the entire site.

A more targeted private-area rule can keep admin paths out while leaving public pages available.

Do not copy another website’s robots file. Its platform, paths and privacy requirements may be different.

Check Googlebot separately from indexing

Googlebot crawling is only one part of Google Search eligibility.

For an important page, confirm:

  • Googlebot is not blocked by robots.txt;
  • the server returns a successful status;
  • there is no unintended noindex;
  • the canonical points to the intended URL;
  • internal links can reach the page;
  • the page is included in the sitemap where appropriate;
  • visible content matches structured data;
  • Google Search Console can inspect the URL.

Google says the normal Search foundations apply to supporting links in AI Overviews and AI Mode. A special “AI schema” does not replace crawl and indexing eligibility.

OpenAI separates its search crawler from some other OpenAI user agents.

OpenAI’s publisher guidance says sites that want public content available for ChatGPT search summaries and snippets should allow OAI-SearchBot and its published IP ranges.

That gives you two places to inspect:

  1. robots.txt rules;
  2. firewall, CDN or bot-management rules.

A security product can challenge or block a crawler even when robots.txt allows it.

Do not disable useful security protections globally. Create the narrowest rule that permits the verified crawler behaviour you intend to support.

Check PerplexityBot independently

Perplexity publishes its own crawler guidance and says PerplexityBot follows robots.txt.

Treat it as a separate user agent. A rule written for Googlebot does not automatically describe what should happen to PerplexityBot.

Review:

  • explicit PerplexityBot rules;
  • wildcard rules that also apply;
  • CDN bot classifications;
  • rate limits;
  • origin-server logs where available.

If you deliberately block PerplexityBot, document that decision. Do not later diagnose missing Perplexity visibility as a content-writing problem without checking access first.

Make the main content available without fragile interaction

A crawler should not have to click an accordion, accept a marketing popup or complete a form to learn what the page offers.

Important service information should exist in the rendered page content.

JavaScript websites can still be crawlable, but fragile client-only rendering creates more failure points. Test the final HTML and rendered page rather than assuming the framework handles it.

Keep these elements accessible:

  • page title;
  • H1;
  • service description;
  • key evidence;
  • location or service area;
  • relevant internal links;
  • contact or next-step link.

Check bot protection after infrastructure changes

Crawler access can break after a website update even when nobody edits the content.

Retest after:

  • moving DNS;
  • enabling Cloudflare bot protection;
  • changing hosting;
  • installing a web application firewall;
  • enabling geographic restrictions;
  • adding rate limiting;
  • moving to a new JavaScript framework;
  • changing robots.txt generation;
  • adding an SEO plugin.

A 403, challenge page or endless redirect is an infrastructure problem before it is a content problem.

Do not use fake crawler user agents as proof

Changing your browser’s user-agent string to “Googlebot” does not prove Google can access the page. Security systems may validate IP ranges or other signals.

Use platform tools and logs where available:

  • Search Console URL Inspection for Google;
  • server or CDN logs;
  • official crawler IP information;
  • robots.txt testing and direct requests;
  • actual AI search observations after access is corrected.

The goal is evidence, not a screenshot of a browser pretending to be a bot.

Check canonical and duplicate signals

A crawler may reach a page but treat another URL as the preferred version.

Look for duplicate variants caused by:

  • HTTP and HTTPS;
  • www and non-www;
  • trailing slash differences;
  • tracking parameters;
  • print pages;
  • duplicate location pages;
  • staging domains.

Use redirects and canonical tags consistently. Do not canonicalise a unique service page to the homepage simply because both mention the company.

Finish with a three-engine access log

For each commercial page, keep one row:

URL Google OAI-SearchBot PerplexityBot Index/directive issue Infrastructure issue Action

That sheet turns “AI cannot find us” into a technical question you can test.

If all three can access the page and the public content is accurate, move to the next layer: entity consistency, evidence, third-party mentions and prompt measurement.

IDJoy includes crawler and page-access checks in its AI search visibility work before recommending content changes.