The industry is starting at the wrong end
The AI visibility conversation has moved quickly to advanced tactics: schema, entity coverage, answer formatting, citation strategy, and model-specific optimization.
Those things can matter. But they come after a more basic question: when a crawler requests the page, does the useful content arrive?
We tested the homepages of 1,000 domains sampled from the Tranco top million. Of the 576 that returned HTML, 100 failed our first test of extractability: they contained fewer than 50 words in the response body. That left 476 pages for the primary score analysis.
This is not evidence that those sites can never appear in an AI answer. It is evidence that many domains have a delivery problem before they have an optimization problem.
The practical takeaway: test the response an AI crawler receives before spending time polishing the content for ranking or citation.
What the crawl actually found
We made one homepage request per domain, followed robots.txt, and did not execute JavaScript. The crawl ran for about 30 minutes on September 5, 2026.
The raw crawler outcomes were:
| Raw outcome | Count | Share |
|---|---|---|
Audit completed successfully (ok) | 501 | 50.1% |
| Crawler marked the response unreadable | 75 | 7.5% |
| Research crawler disallowed by robots.txt | 43 | 4.3% |
| Fetch failed | 381 | 38.1% |
The first row is a crawler outcome, not the final analysis population. Once we applied the pre-specified 50-word minimum to every HTML response, 476 pages qualified for the score analysis and 100 of 576 did not.
That distinction matters. Calling all 501 successful audits “usable content” would blur the raw collection result with the later analysis rule.
The 381 fetch failures also need context. They include dead domains, timeouts, bot protection, and ad-tech or infrastructure domains that were never designed to serve a readable homepage. The study does not show that half the web is broken.
The tighter and more useful finding is the one based on returned HTML:
| Minimum body text | Pages below threshold | Share of 576 HTML responses | 95% CI |
|---|---|---|---|
| 50 words | 100 | 17.4% | 14.5–20.7% |
| 100 words | 114 | 19.8% | 16.7–23.2% |
| 200 words | 142 | 24.7% | 21.3–28.3% |
Even at the permissive 50-word threshold, roughly one in six HTML responses did not provide enough text for the primary analysis.
SEO explains only 7% of extractability
We scored the 476 qualifying pages in two ways: against traditional SEO checks and against checks for retrieving and reusing the served content without rendering JavaScript.
The scores were related, but not closely. Spearman’s ρ was 0.264 (95% CI 0.175–0.351). Squaring that correlation gives a rough shared-variance estimate of about 7%.
That does not make SEO irrelevant. It means a conventional SEO score and this extractability score are measuring substantially different things. A page can have the titles, headings, metadata, and structure expected by an SEO audit while still giving a non-rendering crawler little useful material.
The relationship remained weak at the 100- and 200-word thresholds and across all four Tranco rank bands.
Crawler access separates similar pages by 26.3 points
Among the 476 qualifying pages, 28 had an AI bot crawlability component below 100. That means at least one named answer-time AI crawler was blocked in robots.txt.
Those 28 pages had a mean SEO score of 72.8. The other 448 averaged 74.5—a difference of just 1.7 points.
Their AI visibility scores were much farther apart: 44.8 for the restricted group and 71.1 for the fully allowed group, a difference of 26.3 points.
This is a descriptive comparison, not proof that changing robots.txt will cause a 26.3-point gain. The restricted subgroup is small, and the study was not designed to isolate a causal effect. What the comparison does show is that two groups can look nearly identical to a traditional SEO score and very different to a system that includes crawler access.
One row captures the problem
In this point-in-time research fetch, nytimes.com returned more than 1.1 MB of HTML, 1,112 body words, an H1, and JSON-LD. Its SEO score was 72.
Its AI visibility score was 25, including an AI bot crawlability component of 25.
That row does not tell us whether the New York Times appears in AI answers. It shows the narrower point this study was built to test: browser-visible content, conventional SEO signals, and crawler-level accessibility are not interchangeable.
Browser-visible does not mean crawler-readable
Of the 75 responses the crawler marked unreadable, 33 contained enough HTML scaffolding to suggest that client-side rendering may explain the missing text. That is 44.0% (95% CI 33.3–55.3%).
The content may have appeared after JavaScript ran in a browser. It was not present in the initial response our non-rendering crawler received.
This is why screenshots and manual page checks are not enough. The relevant diagnostic is the actual response delivered to the crawler being tested.
Other observations from the 476 qualifying pages:
- 52.3% included JSON-LD (249 pages; 95% CI 47.8–56.8%).
- The median page contained 855 body words (p25 411, p75 1,477).
- Mean AI visibility varied little by popularity band: 71.9, 67.5, 69.6, and 69.2 from the highest- to lowest-ranked strata.
What the study does not show
- It does not measure citation or source selection. Extractability is a prerequisite and a proxy, not a guarantee that an AI engine will choose or cite a page.
- It does not represent every page on a domain. We tested homepages only, and homepages often depend more heavily on JavaScript than editorial or product pages.
- It does not recreate every crawler. We made one request with a disclosed research user agent, followed robots.txt, and did not execute JavaScript.
- It does not turn proprietary scores into objective facts. The observable signals—response status, body text, HTML size, JSON-LD, and robots rules—are the primary evidence. The composite scores provide supporting context.
- It does not make fetch failures random. Domains behind bot protection or other access controls may differ systematically from the pages we could score.
What to check before optimizing for AI
Start with delivery. For the page and crawler you care about, check:
- Does the request return HTML? Record the status code, redirects, timeout, and any bot challenge.
- Is the main content in the initial response? Compare the raw HTML with the rendered browser view.
- Does robots.txt allow the relevant crawler? Different AI products use different crawler identities and controls.
- Is there enough coherent body text to extract? A shell, consent screen, or navigation-only response is not the article or product page you intended to publish.
- Only then assess optimization. Review structure, entities, schema, evidence, answer clarity, and internal linking once the content reliably reaches the machine.
Method and open data
The full research report brings the analysis, delivery breakdowns, failure patterns, and nine exhibits together in one downloadable PDF. The technical companion publishes the protocol, interpretation guidance, and planned-versus-applied method notes.
The full 1,000-row CSV retains every request, including failures. Word count, HTML size, JSON-LD presence, and robots.txt rules are observable signals readers can re-derive; columns prefixed aiomapper_ belong to our measurement instrument. The sample came from Tranco list 5674N, generated August 30, 2026. The 50-word lower bound was borrowed from the document-length filter described in the Gopher paper; it is a deliberately permissive classification rule, not a universal definition of quality.
Want to inspect your own page at the same starting line? The free checker shows what a non-rendering crawler can retrieve, with no signup required.