# ══════════════════════════════════════════════════════════════ # DEVONTIX SOLUTIONS — robots.txt # https://devontixsolutions.com # # Policy: this site is FULLY OPEN to search engines, AI answer # engines and AI training crawlers. We want to be found, quoted # and cited. Nothing on this site is gated. # # Machine-readable summary for LLMs: /llms.txt # Full text corpus for LLMs: /llms-full.txt # AI usage policy: /ai.txt # Last reviewed: 2026-08-10 # ══════════════════════════════════════════════════════════════ # ── DEFAULT: EVERY CRAWLER, EVERYTHING ──────────────────────── User-agent: * Allow: / Disallow: # ══════════════════════════════════════════════════════════════ # TRADITIONAL SEARCH ENGINES # ══════════════════════════════════════════════════════════════ User-agent: Googlebot Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-News Allow: / User-agent: Googlebot-Video Allow: / User-agent: GoogleOther Allow: / User-agent: GoogleOther-Image Allow: / User-agent: Google-InspectionTool Allow: / User-agent: Storebot-Google Allow: / User-agent: AdsBot-Google Allow: / User-agent: AdsBot-Google-Mobile Allow: / User-agent: APIs-Google Allow: / User-agent: Mediapartners-Google Allow: / User-agent: bingbot Allow: / User-agent: adidxbot Allow: / User-agent: msnbot Allow: / User-agent: BingPreview Allow: / User-agent: MicrosoftPreview Allow: / User-agent: Slurp Allow: / User-agent: DuckDuckBot Allow: / User-agent: Baiduspider Allow: / User-agent: YandexBot Allow: / User-agent: Yeti Allow: / User-agent: SeznamBot Allow: / User-agent: PetalBot Allow: / User-agent: Qwantify Allow: / User-agent: MojeekBot Allow: / User-agent: Applebot Allow: / # ══════════════════════════════════════════════════════════════ # AI ANSWER ENGINES & AI ASSISTANTS (live retrieval) # These fetch pages to answer user questions in real time. # Being allowed here is how we appear inside AI answers. # ══════════════════════════════════════════════════════════════ # ─ OpenAI / ChatGPT ─ User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # ─ Anthropic / Claude ─ User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-Web Allow: / # ─ Perplexity ─ User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # ─ Google Gemini / AI Overviews grounding ─ User-agent: Google-Extended Allow: / User-agent: Google-CloudVertexBot Allow: / # ─ Microsoft Copilot ─ User-agent: CopilotBot Allow: / # ─ Apple Intelligence ─ User-agent: Applebot-Extended Allow: / # ─ Meta AI ─ User-agent: meta-externalagent Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: meta-externalfetcher Allow: / User-agent: Meta-ExternalFetcher Allow: / User-agent: FacebookBot Allow: / # ─ Mistral ─ User-agent: MistralAI-User Allow: / # ─ xAI / Grok ─ User-agent: xAI-Bot Allow: / User-agent: Grokbot Allow: / # ─ DuckDuckGo AI ─ User-agent: DuckAssistBot Allow: / # ─ You.com ─ User-agent: YouBot Allow: / # ─ Amazon (Alexa / Rufus / Nova) ─ User-agent: Amazonbot Allow: / # ─ Cohere ─ User-agent: cohere-ai Allow: / User-agent: cohere-training-data-crawler Allow: / # ─ Other retrieval + agent crawlers ─ User-agent: FirecrawlAgent Allow: / User-agent: Diffbot Allow: / User-agent: DiffbotBot Allow: / User-agent: Bytespider Allow: / User-agent: TikTokSpider Allow: / User-agent: Timpibot Allow: / User-agent: Webzio-Extended Allow: / User-agent: ProRataInc Allow: / User-agent: Andibot Allow: / User-agent: Brightbot Allow: / User-agent: SBIntuitionsBot Allow: / User-agent: PanguBot Allow: / User-agent: YisouSpider Allow: / # ══════════════════════════════════════════════════════════════ # AI TRAINING / DATASET CRAWLERS # Allowed on purpose. We want this studio represented in the # models that people will ask for agency recommendations. # ══════════════════════════════════════════════════════════════ User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / User-agent: CCBot Allow: / User-agent: AI2Bot Allow: / User-agent: AI2Bot-Dolma Allow: / User-agent: ImagesiftBot Allow: / User-agent: Omgilibot Allow: / User-agent: Omgili Allow: / # ══════════════════════════════════════════════════════════════ # SOCIAL / LINK PREVIEW UNFURLERS # Needed for rich cards when our links get shared. # ══════════════════════════════════════════════════════════════ User-agent: facebookexternalhit Allow: / User-agent: Twitterbot Allow: / User-agent: LinkedInBot Allow: / User-agent: Slackbot Allow: / User-agent: Slackbot-LinkExpanding Allow: / User-agent: Discordbot Allow: / User-agent: TelegramBot Allow: / User-agent: WhatsApp Allow: / User-agent: Pinterestbot Allow: / User-agent: redditbot Allow: / User-agent: SkypeUriPreview Allow: / User-agent: Embedly Allow: / User-agent: Iframely Allow: / # ══════════════════════════════════════════════════════════════ # SEO / MONITORING TOOLS — allowed, but throttled so they never # affect real-visitor performance. # ══════════════════════════════════════════════════════════════ User-agent: AhrefsBot Allow: / Crawl-delay: 10 User-agent: SemrushBot Allow: / Crawl-delay: 10 User-agent: rogerbot Allow: / Crawl-delay: 10 User-agent: dotbot Allow: / Crawl-delay: 10 User-agent: MJ12bot Allow: / Crawl-delay: 10 User-agent: DataForSeoBot Allow: / Crawl-delay: 10 User-agent: Screaming Frog SEO Spider Allow: / User-agent: Chrome-Lighthouse Allow: / User-agent: archive.org_bot Allow: / User-agent: ia_archiver Allow: / # ══════════════════════════════════════════════════════════════ # OPTIONAL BLOCK LIST — intentionally left disabled. # Uncomment a block only if a crawler becomes a real problem. # Blocking AI crawlers removes you from AI answers, so think # hard before enabling any of this. # ══════════════════════════════════════════════════════════════ # # User-agent: SemrushBot # Disallow: / # # User-agent: Bytespider # Disallow: / # ══════════════════════════════════════════════════════════════ # SITEMAPS # ══════════════════════════════════════════════════════════════ Sitemap: https://devontixsolutions.com/sitemap.xml Sitemap: https://devontixsolutions.com/sitemap-pages.xml Sitemap: https://devontixsolutions.com/sitemap-images.xml