# goodbotbad.bot

# The archive is meant to be found: the facet pages in docs/01 §09 are the entire
# long-tail search surface, so nothing under /p/, /c/, /m/, /tag/ or /verdict/ is
# disallowed here.
#
# What is excluded is everything that is not content — the app's own surfaces, which also
# answer `X-Robots-Tag: noindex, nofollow` (`NotContent`). Belt and braces, because a crawler
# that ignores one should still be stopped by the other. A header and not a `<meta>` tag,
# because most of what those paths return is a redirect or JSON, and neither carries markup.

# **The signals live here rather than at the edge.** Cloudflare offers a managed
# robots.txt, and while it was on, this file was not the policy: the served result carried
# two `User-agent: *` groups — Cloudflare's `Allow: /` above ours — and an allow and a
# disallow of equal specificity resolve to the least restrictive, so this repository's
# `Disallow` lines were being overridden by a file that is not in it. It is off at
# the zone now, and this is the only robots.txt the site has.
#
#   search    building a search index and returning results. Yes; the long tail is the plan.
#   ai-input  grounding an answer in this content — RAG, AI search summaries. Yes, and
#             deliberately more permissive than the managed default, which said nothing at
#             all. An archive with no audience is found by being cited, and a citation is a
#             reader who now knows the site exists.
#   ai-train  training or fine-tuning on it. **No — and this is the one the site actually
#             has standing to refuse.** The transcripts are quoted material it hosts rather
#             than owns; what is originally ours is the layer on top, the titles and notes
#             and rulings and the pair axis, and that is the part worth reserving.
#   use       reference rather than full: cite it and link to it, do not republish it.
#
# An express reservation under Article 4 of EU Directive 2019/790, which is what makes this
# a reservation of rights rather than a request.
User-agent: *
Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference
Disallow: /settings
Disallow: /submit
Disallow: /submissions/
Disallow: /d/
Disallow: /moderation
Disallow: /mcp
Disallow: /api/
Disallow: /login

# The training crawlers by name, because a signal states a position and a rule enforces
# one. A named group replaces the wildcard group for that agent rather than merging with
# it, so each of these is shut out entirely rather than inheriting the paths above.
#
# **None of this touches how agents actually reach the site.** The write path is the MCP
# server — an authenticated POST to `/mcp`, not a crawl — so blocking `ClaudeBot` costs
# nothing the product depends on. The search-side crawlers these same vendors run under
# other names are deliberately absent: those are the ones that put a citation in front of
# a reader, and `ai-input=yes` above invites exactly that.
User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

Sitemap: https://goodbotbad.bot/sitemap.xml