Broad crawls and scaling out

A broad crawl visits many domains, often without knowing them in advance. Scrapy's defaults are tuned for crawling a single site, so the bottlenecks move from politeness to throughput and memory. The optimization guide recommends:

• Raise global concurrency: set CONCURRENT_REQUESTS close to CONCURRENT_REQUESTS_PER_DOMAIN times the number of domains in flight, as CPU and memory allow, and raise SCRAPER_SLOT_MAX_ACTIVE_SIZE when more concurrency stops helping.
• Keep the scheduler domain-aware: SCHEDULER_PRIORITY_QUEUE = "scrapy.pqueues.DownloaderAwarePriorityQueue" (the default in current versions; older ones needed it set) picks requests for domains with free download slots, so one slow site can't dominate.
• Speed up DNS with a local caching resolver and a larger REACTOR_THREADPOOL_MAXSIZE, since DNS lookups use threads.
• Cut the cost of bad responses: disable or reduce retries, lower DOWNLOAD_TIMEOUT, set REDIRECT_ENABLED = False unless you need redirects, and cap sizes with DOWNLOAD_MAXSIZE.
• Consider breadth-first order, DEPTH_PRIORITY = 1 with FIFO queues (SCHEDULER_DISK_QUEUE = "scrapy.squeues.PickleFifoDiskQueue", SCHEDULER_MEMORY_QUEUE = "scrapy.squeues.FifoMemoryQueue"): it reaches many sites shallowly before going deep, and the docs suggest trying it when memory is the bottleneck. JOBDIR keeps the queue on disk.
• Guard memory and traps with MEMUSAGE_LIMIT_MB and DEPTH_LIMIT (infinite calendars and session-id URLs are classic traps).

When one process isn't enough, scale out: run several Scrapy processes sharing a central queue and dupefilter, typically with the scrapy-redis extension, or partition the URL space (by domain hash) across independent crawlers.

At this scale, politeness is still per site: keep obeying robots.txt and per-domain limits.