Design a Web Crawler

Intermediate · 25 min read · High-Level Design, Case Studies: Core

URL frontier, politeness, deduplication and recrawl scheduling.

Design a web crawler that downloads billions of pages for a search engine: start from seed URLs, fetch pages, extract links, and keep going — politely, without crawling the same page twice, and while keeping important pages fresh.

1. Clarify the problem

2. Requirements

Functional: given seeds, crawl the web; extract and follow links; store page content and metadata; recrawl over time.

3. Capacity estimation

So: the fetch rate per machine is limited by network latency, not CPU (thousands of concurrent connections per machine with async I/O). Content goes to object storage. URL dedup needs a compact, partitioned structure.

4. High-level design

Crawl loop

5. Deep dives

Seeds determine how quickly the crawl reaches the useful web. Good strategies: partition the URL space by locality (popular sites per country) or by topic (news, shopping, sports), start from high-quality hubs (major directories, sitemaps, sites with many outlinks), and keep adding seeds from new domain registrations…

6. Data stores

7. Trade-offs summary

8. What interviewers look for

Interview questions

  • Why is a simple BFS queue not enough for a production crawler?
  • How do you avoid hitting the same website with too many requests when you have 1,000 fetcher threads?
  • How would you detect that two pages are near-duplicates?

Practice

Write your answer before opening the suggested one. Anything you mark "Partly" or "Missed it" comes back in your review deck.

Further reading

Key takeaways

  • The frontier is the heart of the crawler: priority (front queues) plus politeness (per-host back queues and a next-fetch heap).
  • Normalise URLs and dedup them with Bloom filters or fingerprint stores; dedup content with hashes and SimHash.
  • Respect robots.txt and back off on slow or overloaded hosts.
  • Defend against spider traps with depth, length and per-host limits.
  • Partition by host, store content in object storage, and schedule recrawls by estimated change rate.

Previous: Design Search Autocomplete · Next: Design a News Feed