# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
# 🤖 PROMETHEUS SEO ENGINE - ROBOTS.TXT
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
#
# Intelligent crawler rules that maximize SEO while protecting sensitive data.
# This file tells search engines what they can and cannot index.
#
# ARCHITECTURE:
# - Allow all major search engines (Google, Bing, Yahoo, DuckDuckGo)
# - Block aggressive bots and scrapers
# - Protect private pages (admin, auth, API)
# - Rate limit to prevent server overload
# - Provide sitemap location for indexing
#
# @author THE ARCHITECT (NASA-Level Engineering)
# @version 1.0.0 - INDUSTRY LEADING
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

# ═══════════════════════════════════════════════════════════════════
# UNIVERSAL RULES (All Bots)
# ═══════════════════════════════════════════════════════════════════

User-agent: *

# ─────────────────────────────────────────────────────────────────
# ALLOWED PATHS (Let search engines index these)
# ─────────────────────────────────────────────────────────────────

Allow: /
Allow: /store/
Allow: /store/products/
Allow: /user/
Allow: /podcast/
Allow: /music/
Allow: /groups/
Allow: /share/
Allow: /pray/

# ─────────────────────────────────────────────────────────────────
# BLOCKED PATHS (Keep these private)
# ─────────────────────────────────────────────────────────────────

# Authentication & Security
Disallow: /login
Disallow: /register
Disallow: /reset-password
Disallow: /verify-email

# Admin & Dashboard
Disallow: /admin/
Disallow: /admin-store
Disallow: /admin-podcasts
Disallow: /admin-dashboard
Disallow: /commission-dashboard
Disallow: /seller-wallet
Disallow: /data-warehouse-dashboard

# Private User Pages
Disallow: /edit-profile
Disallow: /edit-external-links

# API Endpoints (don't index these)
Disallow: /api/

# Internal Pages
Disallow: /search
Disallow: /discover-users

# Payment & Checkout (sensitive)
Disallow: /store-payment-success
Disallow: /checkout

# Testing & Dev (if any)
Disallow: /test
Disallow: /dev
Disallow: /_next/
Disallow: /static/

# ─────────────────────────────────────────────────────────────────
# FILE TYPES TO IGNORE
# ─────────────────────────────────────────────────────────────────

Disallow: /*.json$
Disallow: /*.xml$
Disallow: /*.js$
Disallow: /*.css$
Disallow: /*.map$

# ═══════════════════════════════════════════════════════════════════
# GOOGLEBOT SPECIFIC RULES (The Most Important Bot)
# ═══════════════════════════════════════════════════════════════════

User-agent: Googlebot
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /login

# Allow Google to index images (important for image search)
User-agent: Googlebot-Image
Allow: /images/
Allow: /uploads/
Allow: /assets/

# Allow Google to index videos (for video search)
User-agent: Googlebot-Video
Allow: /videos/
Allow: /uploads/

# ═══════════════════════════════════════════════════════════════════
# BINGBOT RULES (Microsoft Search)
# ═══════════════════════════════════════════════════════════════════

User-agent: Bingbot
Allow: /
Disallow: /admin/
Disallow: /api/
Crawl-delay: 1

# ═══════════════════════════════════════════════════════════════════
# SOCIAL MEDIA BOTS (For Rich Previews)
# ═══════════════════════════════════════════════════════════════════

# Facebook/Meta Bot
User-agent: facebookexternalhit
Allow: /
Allow: /store/products/
Allow: /user/
Allow: /podcast/
Allow: /music/
Allow: /share/

# Twitter Bot (X Bot)
User-agent: Twitterbot
Allow: /
Allow: /store/products/
Allow: /user/
Allow: /podcast/
Allow: /music/
Allow: /share/

# WhatsApp Bot
User-agent: WhatsApp
Allow: /
Allow: /store/products/
Allow: /user/
Allow: /share/

# Telegram Bot
User-agent: TelegramBot
Allow: /
Allow: /store/products/
Allow: /user/
Allow: /share/

# LinkedIn Bot
User-agent: LinkedInBot
Allow: /
Allow: /store/products/
Allow: /user/
Allow: /share/

# ═══════════════════════════════════════════════════════════════════
# BAD BOTS (Block Aggressive Scrapers)
# ═══════════════════════════════════════════════════════════════════

# Block known bad bots
User-agent: MJ12bot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: YandexBot
Crawl-delay: 5

# Block AI training scrapers (protect your content)
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Claude-Web
Disallow: /

# ═══════════════════════════════════════════════════════════════════
# SITEMAP LOCATION (THE MOST IMPORTANT LINE!)
# ═══════════════════════════════════════════════════════════════════

# Tell search engines where to find your sitemap
Sitemap: https://sanctuaryfaith.org/sitemap.xml
Sitemap: https://sanctuaryfaith.org/sitemap-images.xml
Sitemap: https://sanctuaryfaith.org/sitemap-videos.xml
Sitemap: https://sanctuaryfaith.org/prayers-sitemap.xml

# ═══════════════════════════════════════════════════════════════════
# CRAWL RATE LIMITS (Prevent Server Overload)
# ═══════════════════════════════════════════════════════════════════

# Default crawl delay for all bots (1 second between requests)
Crawl-delay: 1

# ═══════════════════════════════════════════════════════════════════
# NOTES FOR FUTURE
# ═══════════════════════════════════════════════════════════════════

# This robots.txt is designed to:
# 1. ✅ Allow indexing of all public content
# 2. ✅ Block private/sensitive pages
# 3. ✅ Enable social media rich previews
# 4. ✅ Prevent aggressive scraping
# 5. ✅ Protect user data and admin areas
# 6. ✅ Provide sitemap for efficient indexing
# 7. ✅ Rate limit to protect server resources

# Update this file when you add:
# - New admin pages
# - New API endpoints
# - New private sections
# - New public content areas

# ═══════════════════════════════════════════════════════════════════
# END OF FILE
# ═══════════════════════════════════════════════════════════════════