# Production robots policy. Staging deploys are protected by the
# X-Robots-Tag: noindex header set in netlify.toml — that header, not this
# file, is what keeps staging out of indexes.

# Default: every crawler may fetch every public page. Non-public surfaces are
# kept out of indexes with a noindex response header, never a Disallow rule
# here — a Disallow line in this public file only advertises the path it hides
# (see netlify.toml for that header policy).
User-agent: *
Allow: /

# AI-crawler policy (owner-decided 2026-07-04): WELCOME ALL. This is a
# discovery-first marketing site, so being cited by AI answer engines and
# included in model training both widen reach. The groups below make that
# welcome explicit for the major AI crawlers; any AI crawler not named here is
# still allowed by the wildcard group above. (Google-Extended governs only
# Gemini training/grounding — AI Overviews rides normal Google Search indexing
# and would be unaffected either way. The welcome-all policy stands on reach,
# not on that fear.)

# AI answer / search engines — fetch pages to cite in live answers.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# AI model-training crawlers.
User-agent: GPTBot
User-agent: Google-Extended
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: CCBot
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: Bytespider
Allow: /

Sitemap: https://www.viduramziai.lt/sitemap-index.xml