# robots.txt for www.aice-lab.bd
# Source of truth: layouts/robots.txt (Hugo)
#
# Cloudflare prepends an "AI Audit Content Signals" block at the edge with
# the default policy: search=yes, ai-train=no. That policy stands. AI bots
# that CF lists (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider,
# Applebot-Extended, Amazonbot, meta-externalagent, etc.) are blocked from
# training; the default User-agent: * with Content-Signal allows search
# indexing and AI answer generation (RAG / "ai-input").
#
# We layer ON TOP only with disallows for aggressive SEO scrapers — they
# consume bandwidth without adding value, and CF doesn't filter them by
# default. We do NOT add explicit User-agent blocks for search engines or
# AI crawlers that CF already addresses, to avoid override semantics.

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: BLEXBot
Disallow: /

Sitemap: https://www.aice-lab.bd/sitemap.xml