Publish your terms for AI bots
Create a hosted policy page that explains what crawlers may do with your content and where licensing inquiries should go.
AI crawler policy control
Control how AI bots use your website.
Generate crawler rules, publish AI-use permissions, and prepare your content for licensing or blocking.
Publishes clear permissions and terms for crawlers to reference.
Generates robots.txt, llms.txt, and policy-page copy you can review.
Prepares a path for licensing inquiries and paid crawler access.
Generated output
Use these snippets as a starting point, then publish a canonical policy page.
3Policy surfaces
6Permission fields
1Canonical URL
What BotConsent sells
From free snippets to monitored crawler policy operations.
BotConsent is not a magic shield. It gives your site a clear policy layer, deployable technical signals, and an upgrade path for teams that need hosting, audits, and alerts.
Ship crawler-ready snippets
Generate robots.txt, llms.txt, Content Signals-style language, and commercial-use terms that fit your policy.
Know what your current robots.txt says
Use the audit and monitoring flows to spot missing bot groups, conflicting directives, and changes that could affect crawler access.
Pricing
Compare every plan
Clear paid paths after the free generator.
Free
Generate basic policy snippets after email capture.
$0/mo
- AI bot policy generator
- Basic robots.txt output
- Hosted policy preview
Starter
Publish a hosted policy page for one domain.
$19/mo
- Hosted BotConsent policy page
- Version history
- robots.txt and llms.txt snippets
Pro
Monitor policies across multiple domains.
$79/mo
- Multiple domains
- Monitoring and alerts
- Crawler policy checks
Agency
Manage client crawler policies at scale.
$199/mo
- Client workspace
- Reusable templates
- White-label reports for 25 domains
Opening secure Stripe checkout: Plan at price selected.
One-time audit
Get a crawler-readiness report for $49.
The audit is for website owners who want to know what their current crawler policy says before they buy ongoing monitoring.
Readiness report
example.com
72Policy score
Good search coverage. Training and commercial-use terms need clearer signals.
- OK Standard search crawlers are allowed.
- WARN GPTBot and Google-Extended rules are missing.
- WARN No canonical AI policy page found.
- FIX No licensing route for paid crawler access.
FAQ
Plain answers for site owners.
Does BotConsent legally enforce my permissions?
No. BotConsent helps publish preferences, document terms, prepare technical signals, and monitor policy readiness. BotConsent is not legal advice and cannot guarantee crawler compliance.
Can I allow search, block training?
Yes. That is the default policy posture in the generator: allow search indexing and assistant retrieval, while blocking model training and commercial scraping unless you turn them on.
How does this relate to Cloudflare AI Crawl Control?
Cloudflare offers infrastructure-level crawler controls and Pay Per Crawl experiments. BotConsent is independent and helps you draft, publish, and monitor policy language that can complement whatever infrastructure you use.
What about OpenAI crawlers and Google-Extended?
BotConsent includes common crawler tokens such as GPTBot, OAI-SearchBot, ChatGPT-User, and Google-Extended as editable starting points. You should review official crawler documentation before publishing.
Do bad bots respect robots.txt?
Some do, some do not. A published policy is still useful for good-faith operators, partners, internal documentation, and legal review, but evasive scrapers may ignore it.
Can BotConsent help prepare for paid crawler access?
Yes. It helps you publish licensing language, create a canonical inquiry path, and keep policy files consistent while the paid-access market continues to mature.