Expose silent RFC 9309 & firewall blocks in one live crawl
Evaluates longest-match robots.txt rules across 5 search crawlers, sends real OAI-SearchBot & PerplexityBot HTTP probes to catch silent 403/429 WAF blocks, and checks noindex, SSR word count, sitemaps, and JSON-LD — never penalizing blocked training scrapers (GPTBot, ClaudeBot, Google-Extended).