Problem

Design a Distributed Web Crawler

Description

Design a distributed system that crawls billions of web pages efficiently, deduplicates URLs, and respects politeness policies.

Prerequisites