User-agent: *
Allow: /

# Explicit content-usage signals (see https://contentsignals.org), added
# 2026-08-17 in place of Cloudflare's "Managed robots.txt" feature --
# that feature's own dashboard description is "signal that your content
# should NOT be used for AI training", a block-only switch with no way to
# express an explicit allow, and it was overriding this file's own
# intent (auto-injecting a directly contradictory Disallow for the exact
# crawlers explicitly allowed below). Confirmed directly, 2026-08-17, not
# assumed: the Managed-robots.txt-injected block is edge-injected on top
# of whatever this repo serves, not derived from it -- this file was
# never the problem, and turning that toggle off (direct instruction)
# lets this be the actual, single source of truth again.
# search:   allow building a search index / returning results.
# ai-input: allow real-time retrieval/grounding (an agent looking this
#           page up live) -- yes, since that's the literal product.
# ai-train: allow training/fine-tuning on this content -- yes, direct
#           instruction ("I say we allow training"), decided with the
#           one-way-door tradeoff (can't un-train a model later) named
#           and considered, not defaulted into.
Content-Signal: search=yes,ai-input=yes,ai-train=yes,use=full

# Explicit block, direct instruction ("I blocked spyder bot") -- kept
# here rather than relying on Cloudflare's own managed list so it
# survives regardless of that toggle's state. Real, documented reputation
# for aggressive/less-transparent scraping, distinct from the legitimate
# AI-assistant crawlers explicitly allowed below.
User-agent: Bytespider
Disallow: /

# Explicit allows for known AI/LLM crawlers, in addition to the wildcard
# above (which already permits everything) -- added 2026-08-14 as a
# deliberate, documented GEO decision, not to change behavior (the
# wildcard already covers these) but to make the intent explicit rather
# than an accidental default. See /llms.txt for a structured summary
# aimed specifically at LLM/agent readers.
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: CCBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://syntology.ai/sitemap.xml