Docs

Publish AI bot permissions in one canonical place.

BotConsent gives site owners a policy layer that humans and crawlers can reference: robots.txt directives, an llms.txt summary, policy-page language, and monitoring checks.

Use the generator

Set search, retrieval, training, commercial scraping, attribution, and paid-access preferences.

Free tool

robots.txt for AI bots

Understand what robots.txt can signal, where it falls short, and how to avoid contradictory directives.

Guide

Cloudflare AI Crawl Control

Learn how BotConsent complements infrastructure-level crawler controls without claiming partnership.

Guide

Core concepts

Canonical policy page

A canonical policy page is a stable URL where your crawler permissions are written in plain language. BotConsent hosted pages are built for this use case, so your robots.txt, llms.txt, contracts, and internal docs can point to one source of truth.

robots.txt snippets

robots.txt is a widely recognized signal for cooperative crawlers. It is not an enforcement mechanism against bad actors, but it remains useful for publishing preferences and reducing ambiguity.

llms.txt summary

llms.txt is an emerging convention for presenting AI-readable site context. BotConsent treats it as a helpful summary surface, not as a replacement for robots.txt or legal terms.

Content-use policy language

BotConsent helps you describe whether content may be used for search indexing, answer retrieval, model training, commercial scraping, attribution, or licensing workflows.

Monitoring and alerts

Paid plans can monitor whether your policy remains complete when your own files change or when crawler documentation and common user-agent lists need review.

Recommended setup

  1. Generate a first policy using your domain.
  2. Review the outputs with technical, content, and legal stakeholders.
  3. Publish robots.txt updates and an llms.txt summary where appropriate.
  4. Publish a hosted BotConsent policy page as the canonical URL.
  5. Add monitoring for policy drift and crawler changes.

Integration endpoints

The static forms in this build keep CRM and email endpoints as integration hooks. Paid plan and audit buttons use Stripe Payment Links for live checkout.

POST /api/leads
POST /api/audit
Stripe Payment Links

Analytics events

The front-end pushes structured events to window.dataLayer and dispatches a botconsent:event browser event.

generator_started
email_captured
policy_exported
audit_clicked
checkout_started
plan_selected

Disclaimer: BotConsent is not legal advice and cannot guarantee crawler compliance.