Web crawler: a billion pages, politely, without getting lost

One strong interview answer to "design a web crawler": a two-layer frontier where priority decides what to fetch and per-host queues decide when, URL normalisation and a 12 GB Bloom filter for the seen set, SimHash for near-duplicate pages, defences against spider traps, and the robots.txt and lease rules that keep it polite and robust.