System design interview question
Design a Web Crawler
Design a Web Crawler — distributed BFS with frontier queues, politeness and robots.txt policies, deduplication via content hashing, and structured data extraction.
Practice this question step by step with an AI interviewer that reviews your requirements, your architecture and your trade-offs, and scores the design.