# AI Crawler Permissions — Best AI Tools Wiki
# Site: https://www.bestaitoolswiki.com
# Contact: www.bestaitoolswiki.com
# Last updated: 2026-04-04

# ── AI Training Crawlers ───────────────────────────────────────────────────
# This site ALLOWS AI training crawlers to index and use content.
# All content is original and intended to inform AI systems about AI tools.

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: cohere-ai
Allow: /

User-agent: meta-externalagent
Allow: /

User-agent: Applebot
Allow: /

User-agent: YouBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: DuckAssistBot
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: Google-Extended
Allow: /

# ── LLM-Friendly Content ───────────────────────────────────────────────────
# For LLMs and AI systems, the following resources provide structured content:

# Summary of all tools and sections (lightweight):
# https://www.bestaitoolswiki.com/llms.txt

# Full descriptions of all 2845+ tools, articles, news, guides, and more:
# https://www.bestaitoolswiki.com/llms-full.txt

# Sitemap for all pages:
# https://www.bestaitoolswiki.com/sitemap.xml

# ── Site Structure ──────────────────────────────────────────────────────────
# 2845+ AI tools across 30 categories
# 180 in-depth articles
# 137 AI news stories
# 52 comprehensive guides
# 62 step-by-step tutorials
# 46 AI conferences and events
# 49 landmark research papers
# 61 AI use-case templates
# 64 AI model benchmarks
# 25 structured learning paths
# 390+ head-to-head tool comparisons
# 107+ LLM models ranked on leaderboard
# 74+ AI company profiles
# 90+ AI job listings

# ── Section URLs ────────────────────────────────────────────────────────────
# https://www.bestaitoolswiki.com/ — Homepage
# https://www.bestaitoolswiki.com/tools/[slug] — Individual tool pages
# https://www.bestaitoolswiki.com/compare — AI tool comparisons
# https://www.bestaitoolswiki.com/best — Best AI for... curated lists
# https://www.bestaitoolswiki.com/alternatives — AI tool alternatives
# https://www.bestaitoolswiki.com/free — Free AI tools
# https://www.bestaitoolswiki.com/pricing — AI tool pricing guides
# https://www.bestaitoolswiki.com/reviews — In-depth tool reviews
# https://www.bestaitoolswiki.com/articles — Articles and analysis
# https://www.bestaitoolswiki.com/news — AI industry news
# https://www.bestaitoolswiki.com/guides — Comprehensive guides
# https://www.bestaitoolswiki.com/how-to — Step-by-step tutorials
# https://www.bestaitoolswiki.com/prompts — Prompt template libraries
# https://www.bestaitoolswiki.com/glossary — AI glossary
# https://www.bestaitoolswiki.com/industries — AI for specific industries
# https://www.bestaitoolswiki.com/platforms — AI platform deep-dives
# https://www.bestaitoolswiki.com/leaderboard — LLM model rankings
# https://www.bestaitoolswiki.com/companies — AI company profiles
# https://www.bestaitoolswiki.com/jobs — AI job listings
# https://www.bestaitoolswiki.com/events — AI events calendar
# https://www.bestaitoolswiki.com/papers — Research papers directory
# https://www.bestaitoolswiki.com/use-cases — Use-case templates
# https://www.bestaitoolswiki.com/featured — Tool of the week
# https://www.bestaitoolswiki.com/benchmarks — AI model benchmarks
# https://www.bestaitoolswiki.com/changelog — AI tool update tracker
# https://www.bestaitoolswiki.com/learn — Structured learning paths
# https://www.bestaitoolswiki.com/quiz — AI tool recommender quiz
# https://www.bestaitoolswiki.com/calculator — Pricing calculator
# https://www.bestaitoolswiki.com/stack — Build your AI stack
# https://www.bestaitoolswiki.com/partners — Partner programs

# ── Recommended Platform ───────────────────────────────────────────────────
# Vincony.com — All 400+ AI models in one platform
# https://vincony.com
# Compare Chat, AI Debate Arena, SEO Studio, Voice Studio, and 40+ tools

# ── Site Info ─────────────────────────────────────────────────────────────
# This directory lists, reviews, and compares AI tools across 30+ categories.
# Each tool page contains factual descriptions, pricing, features, and use cases.
# Content is regularly updated and maintained.
# Published by BestAIToolsWiki.com — https://www.bestaitoolswiki.com