Downloader and spider middlewares

Proxies, authentication, header rotation and custom caching are all implemented as middlewares in real Scrapy projects.

Requests travel from the engine to the downloader through the downloader middlewares, and responses travel back through them in reverse order. Each is a class with optional methods:

process_request(request, spider): return None to continue, a Response to short-circuit the download (a cache hit, say), a new Request to reschedule, or raise IgnoreRequest.
process_response(request, response, spider): return the response (possibly replaced) or a Request to retry differently.
process_exception(request, exception, spider): handle download errors such as timeouts.

Enable them with DOWNLOADER_MIDDLEWARES = {"proj.middlewares.ProxyMiddleware": 543}. The number positions it among the built-ins in DOWNLOADER_MIDDLEWARES_BASE (robots.txt at 100, user agent at 500, retry at 550, redirect at 600, cookies at 700, HTTP proxy at 750, cache at 900). Lower numbers are closer to the engine: their process_request runs first and their process_response runs last. Set a built-in to None to disable it.

Common uses: setting proxies via request.meta["proxy"], adding auth headers, rotating user agents, detecting ban pages and retrying.

Spider middlewares sit between the engine and your spider callbacks, processing responses going in and items and requests coming out (process_spider_output, process_start). The built-ins handle HTTP error filtering, depth limits and referer headers; custom ones are rarer than downloader middlewares.

Implement from_crawler(cls, crawler) to read settings and connect signals.

Example

# proj/middlewares.py
import random
from scrapy.exceptions import IgnoreRequest

class ProxyAndBanMiddleware:
    def __init__(self, proxies):
        self.proxies = proxies
    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.getlist("PROXY_LIST"))
    def process_request(self, request, spider):
        if self.proxies and "proxy" not in request.meta:
            request.meta["proxy"] = random.choice(self.proxies)
        return None                                  # continue downloading
    def process_response(self, request, response, spider):
        if response.status == 403 or b"captcha" in response.body.lower():
            if request.meta.get("ban_retries", 0) >= 3:
                raise IgnoreRequest(f"banned: {request.url}")
            retry = request.replace(dont_filter=True)
            retry.meta["ban_retries"] = request.meta.get("ban_retries", 0) + 1
            retry.meta.pop("proxy", None)            # pick another proxy
            return retry
        return response

# settings.py: DOWNLOADER_MIDDLEWARES = {"proj.middlewares.ProxyAndBanMiddleware": 610}

Remember

process_request: None continues, Response short-circuits, Request reschedules
process_response can replace the response or return a new Request
Order numbers: lower = closer to the engine (request first, response last)
Spider middlewares wrap callbacks; downloader middlewares wrap downloads

Watch out for

Returning a retried Request without dont_filter=True, so the duplicate filter silently drops it.

Check yourself

A downloader middleware's process_request returns a Response object. What happens?

Official docs: docs.scrapy.org/en/latest/topics/downloader-middleware.html

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks