robots.txt coverage
We review whether common search, AI retrieval, AI training, and commercial crawler groups are addressed.
Crawler readiness audit
Know what your current robots.txt says.
For $49, get a practical report on AI crawler coverage, missing policy signals, conflicts, and the next steps to publish a canonical BotConsent policy page.
Report preview$49 one-time
68Readiness
Clear search rules. AI training and commercial access rules need review.
- OK robots.txt file detected.
- WARN AI training user-agent directives are incomplete.
- WARN llms.txt or policy-page reference not found.
- FIX No licensing inquiry path is published.
What the audit checks
A focused report, not a generic SEO scan.
Policy completeness
We check whether your site explains allowed uses, blocked uses, attribution, and commercial-use terms.
Canonical policy path
We look for a durable policy URL that crawlers, partners, and internal teams can reference.
Licensing readiness
We identify whether bulk commercial use has a clear inquiry path for paid access or licensing.
Order the $49 audit
Submit your domain and contact information, then continue to secure Stripe checkout.
What you receive
- Summary score and top policy gaps
- Current robots.txt interpretation
- AI crawler coverage checklist
- Suggested policy text for search, retrieval, training, scraping, attribution, and licensing
- Recommended next step: free snippets, Starter hosting, Pro monitoring, or Agency workspace
BotConsent does not provide legal advice. The audit is a practical readiness report for technical and business review.