Back to AI crawler checker

Sample AI Crawler Check

robots.txt AI crawler access for NovaDesk — illustrative demo, not a live workspace

www.novadesk.io

https://www.novadesk.io/robots.txt

robots.txt analysis for AI crawlers

Checked 2026/5/28 14:30

100Score
2 pass0 warn0 fail
Validation checklist

Parses live robots.txt — does not simulate per-URL fetches for each bot.

pass

robots.txt reachable

robots.txt returned HTTP 200.

pass

AI crawler access

No blanket Disallow detected for tracked AI crawlers.

AI crawler access

Per-bot status from robots.txt rules (blanket Disallow: /).

GPTBot

OpenAI

pass

ChatGPT-User

OpenAI

pass

OAI-SearchBot

OpenAI

pass

ClaudeBot

Anthropic

pass

PerplexityBot

Perplexity

pass

Google-Extended

Google

pass

CCBot

Common Crawl

pass
robots.txt preview
User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://www.novadesk.io/sitemap.xml

Run a full GEO site audit

Sign in to score your brand homepage across six pillars — technical crawlability, schema, content, citations, and more — no credit card.

Get started free

More free GEO tools

Other standalone checkers — paste a URL, preview results in seconds, no account required.