What is Web Crawler?
Software that systematically fetches web pages by following links — the discovery layer every search and AI system is built on.
Definition
A web crawler is software that systematically fetches pages, follows their links, and feeds what it finds into an index or dataset. Googlebot is the archetype; the AI era added a swarm: training crawlers (GPTBot), search-index crawlers (OAI-SearchBot), and on-demand fetchers (ChatGPT-User) — each with different purposes and rules.
Why it matters
Everything downstream — indexing, retrieval, citations, training presence — starts with a successful crawl. Cloudflare's data shows the stakes rising: crawl volume surged through 2025 with Googlebot up 96% year-over-year and AI bots the fastest-growing class, while sites increasingly gate them. Your crawl posture (who's allowed, what they can reach, what the HTML contains) is now a strategic decision reviewed like a channel budget, not a set-and-forget technical detail.
Frequently asked
How do crawlers discover my pages?
Links (internal and external), sitemaps, and revisits of known URLs. Orphan pages — reachable by no link — often go uncrawled: internal linking is discovery infrastructure.
How is an AI fetcher different from a classic crawler?
Classic crawlers build indexes ahead of need; on-demand fetchers (ChatGPT-User, Perplexity-User) retrieve a specific page the moment a user's question requires it. The second kind is a live buyer moment happening on your server.
Related terms
- AI CitationsThe linked sources an AI assistant credits when generating an answer. Being cited is the AI-era equivalent of ranking #1.
- AI CrawlerBots operated by AI companies — GPTBot, ClaudeBot, PerplexityBot, Google-Extended — that fetch web content for model training or live answer retrieval.
- AI OverviewsGoogle's AI-generated summaries shown above traditional results, now appearing on roughly half of searches and dramatically reducing clicks to websites.
- AI Search VisibilityHow often and how prominently a brand appears in answers generated by AI assistants like ChatGPT, Gemini, Perplexity, and Google AI Overviews.