# JaxSuite robots.txt
#
# ---------------------------------------------------------------------------
# Training crawlers vs. retrieval crawlers — the distinction that matters here
# ---------------------------------------------------------------------------
# TRAINING crawlers (GPTBot, ClaudeBot, Google-Extended) fetch pages in bulk to
# train or ground models. Blocking one costs nothing at query time; it only
# removes us from a future training corpus.
#
# RETRIEVAL crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot,
# Perplexity-User, Claude-SearchBot, Claude-User — plus classic Googlebot and
# Bingbot, which also feed AI Overviews and Copilot) fetch a page *because a
# user just asked a question*. Blocking one of these does not protect anything:
# it deletes JaxSuite from that engine's AI answers, exactly like de-indexing
# from a search engine. Do not block them as a privacy or bandwidth measure.
#
# Every group below carries the identical rule block, so this file is
# behaviourally identical to a single wildcard group. The named
# stanzas exist only to make that permission explicit, self-documenting, and
# reversible on purpose rather than by accident.
#
# ---------------------------------------------------------------------------
# WHEN EDITING: named groups do NOT inherit from `User-agent: *`
# ---------------------------------------------------------------------------
# Per RFC 9309 §2.2.1 a crawler obeys exactly ONE group — the most specific
# user-agent match — and ignores every other group, including the wildcard. A
# rule added only to `User-agent: *` therefore does NOT apply to any bot named
# below. Every Disallow AND every Allow goes in ALL THREE groups: a missing
# Disallow leaves a private area crawlable for the named bots, and a missing
# Allow re-blocks a page the wildcard group exposes. `robots-txt.test.ts`
# asserts both per group, so a partial edit fails CI rather than shipping.
# ---------------------------------------------------------------------------

# ---------------------------------------------------------------------------
# The /api/ exception
# ---------------------------------------------------------------------------
# `Disallow: /api/` keeps crawlers out of the JSON endpoints. The human-readable
# REST reference is the HTML page at /api/v1/docs/ui, which we do want indexed and
# quotable, so the `Allow: /api/v1/docs` prefix below re-opens that subtree. The raw
# OpenAPI JSON at /api/v1/docs stays crawlable (tools and agents fetch it) but is
# served with an `X-Robots-Tag: noindex` header, so only the HTML page is indexed.
# Per RFC 9309 §2.2.2 the longest matching rule wins regardless of order, so the
# narrower `Allow` beats the broader `Disallow`.
# ---------------------------------------------------------------------------
# Why /mfa-verification/ is still listed
# ---------------------------------------------------------------------------
# The 2FA page moved to /auth/2fa/ on 2026-08-25. The old path stays reachable
# as a 308 in next.config.js, and a redirect is a crawlable URL, so both paths
# are disallowed until the legacy redirect is retired.
# ---------------------------------------------------------------------------

# Default: every crawler not named in a group below.
User-agent: *
Allow: /
Allow: /api/v1/docs
Disallow: /dashboard/
Disallow: /auth/
Disallow: /api/
Disallow: /auth/2fa/
Disallow: /mfa-verification/

# Search + live-retrieval crawlers. These decide whether JaxSuite can appear in
# AI answers and search results at all — keep them allowed.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: DuckAssistBot
User-agent: MistralAI-User
Allow: /
Allow: /api/v1/docs
Disallow: /dashboard/
Disallow: /auth/
Disallow: /api/
Disallow: /auth/2fa/
Disallow: /mfa-verification/

# Model-training crawlers. Currently allowed; this stanza makes the existing
# permission explicit so opting out later is a deliberate, reviewable edit.
# Applebot-Extended is training-only: it is the opt-out lever for Apple
# Intelligence and does not affect Applebot's retrieval above.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
Allow: /
Allow: /api/v1/docs
Disallow: /dashboard/
Disallow: /auth/
Disallow: /api/
Disallow: /auth/2fa/
Disallow: /mfa-verification/

Sitemap: https://www.jaxsuite.com/sitemap.xml

# Machine-readable site summary for LLMs (https://llmstxt.org):
# https://www.jaxsuite.com/llms.txt (short) and /llms-full.txt (expanded).