SergeBot
SergeBot is the crawler used by Serge’s website inspection tools. This page documents that technical service, including its user-agent and opt-out controls. Website inspection is separate from ChatGPT campaign preparation and does not guarantee ad eligibility or delivery.
User-Agent string
Every request from SergeBot identifies itself with this exact User-Agent:
SergeBot/1.0 (+https://www.serge.ai/bot; product-findability-scanner)What SergeBot does
SergeBot only runs when a user initiates a scan. It is not an autonomous crawler and does not spider your site. Each scan makes a small, fixed set of requests to well-known paths:
| Resource | Purpose |
|---|---|
/llms.txt | LLM product description |
/llms-full.txt | Extended LLM documentation |
/openapi.json | OpenAPI specification |
/.well-known/agent.json | A2A agent card |
/.well-known/agents.json | Agent directory card |
/robots.txt | Crawler permissions |
/sitemap.xml | Site structure |
/ | Homepage (structured data, JSON-LD) |
/docs, /api, /developers | Developer hub detection |
/pricing | Pricing page detection |
SergeBot also queries external registries (MCP Registry, PulseMCP, npm) for SDK and MCP server presence. These requests do not touch your infrastructure.
What SergeBot does not do
Rate limits
| Limit | Value |
|---|---|
| Max requests per domain per scan | ~20 |
| Max concurrent requests per domain | 6 |
| Per-request timeout | 8 seconds |
| Scan completes in | < 30 seconds |
SergeBot respects robots.txt directives and Crawl-delay values.
Controlling access
User-agent: SergeBot
Allow: /User-agent: SergeBot
Disallow: /Blocking SergeBot can make its website checks inconclusive. Other crawlers have their own identities and rules; this setting does not determine OpenAI account approval or ad delivery.
Data handling
Contact
Questions about SergeBot, false positives, or access issues: