These examples are starting points. BotConsent helps you publish clear preferences, but your team should review policy language before it goes live.
Use when search visibility matters, but archives, analysis, or reporting should not be used for AI model training without a license.
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Policy: https://publisher.com/botconsentUse when product documentation can be indexed for discovery, but commercial scraping and model training should be reviewed first.
Search indexing: Allowed
AI assistant retrieval: Allowed
AI model training: Not allowed
Commercial scraping: Not allowed
Attribution requested: YesUse when client sites need a repeatable policy review process and white-label reports.
Client setup:
- Baseline audit
- Policy approval
- Hosted policy page
- Monthly monitoring
- Report exportUse when you want a clear paid-access path before bulk content use becomes a sales conversation.
Commercial use, dataset creation, or paid crawler access should begin with a licensing inquiry before content is collected.They are not legal guarantees, enforcement tools, or evidence that every crawler will comply. They are clear, reviewable policy starting points that reduce ambiguity for good-faith crawlers, partners, and internal teams.