# robots.txt AI-policy scan — Tranco top 10,000, 2026-07-03

Dataset behind the Sitedex insight "Your robots.txt is a 2023 war memorial"
(https://sitedex.dev/insights/robots-txt-2023-war-memorial). Everything here
is derived from files the sites publish for bots to read; no pages were
crawled.

## Files

- `matrix.jsonl` — one record per scanned domain. Fields: `domain`,
  `fetch_ok`, `usable`, `reason` (failure taxonomy for unusable rows:
  no_dns / fetch_error / bot_challenged / no_robots_txt / html_not_robots /
  other), `status`, `wildcard` (the `*` group's disposition), `blocks_any_ai`,
  `bots` (per-bot `{explicit, effective}` disposition: FULL_BLOCK / PARTIAL /
  EXPLICIT_ALLOW / ABSENT, where `effective` applies wildcard fallback),
  `gaps` (same-vendor sibling analysis: `open` = effectively unrestricted
  siblings, `path_restricted` = siblings with path rules), `stanza_tokens`
  (user-agent tokens observed, capped at 100).
- `wayback.jsonl` — for all 1,197 full-blockers: first quarter each bot token
  was observed in the site's archived robots.txt (Internet Archive; quarterly
  probes attributed to the capture's actual timestamp; first-OBSERVED dates
  are upper bounds on rule age).
- `summary.md` — the full numbers memo, including the methodology, the rigor
  appendix (differential parser validation, stability re-scan, carve-out
  analysis), and a denominator-provenance table for every published figure.

## Method, in one paragraph

We fetched `https://<domain>/robots.txt` (with a `www.` fallback and one
polite retry) for the Tranco top 10,000 on 2026-07-03, identifying as
SitedexBot. This measures DECLARED policy only — edge/WAF enforcement is
invisible to this method, and we never claim a site "blocks AI," only what
its published file says. Sites whose robots.txt was unreadable to an
identified bot (883) are excluded from all rate denominators. Parser
validated against the `robots-parser` reference (99.24% agreement; the
differences are the Disallow-with-carve-outs pattern, documented in
summary.md). Scanner source: https://github.com/wastedcode/sitedex
(tools/robots-scan).

## License

CC BY 4.0. Cite as: Sitedex, "robots.txt AI-policy scan," July 2026,
https://sitedex.dev/insights/robots-txt-2023-war-memorial