TopolearnUnderstanding the data flow lets you place custom logic correctly and debug ordering, duplicates and throughput issues.
Scrapy's core is an event-driven engine on Twisted's reactor (running on asyncio by default in current versions). The data flow for each request:
- The spider's
start()yields initial requests; the engine passes them to the scheduler. - The scheduler checks the dupefilter (by request fingerprint: method, canonical URL, body) and enqueues new requests in priority queues.
Request(priority=10)jumps ahead;dont_filter=Trueskips the duplicate check. - When the downloader has free slots, the engine takes the next request from the scheduler and sends it through the downloader middlewares to the downloader, which manages per-domain slots enforcing concurrency and delays.
- The response returns through the downloader middlewares, then through the spider middlewares into your callback.
- Items the callback yields go to the item pipelines; requests go back to the scheduler.
Everything is non-blocking and runs in one thread. A callback that blocks (a time.sleep, a synchronous database call, heavy CPU work) stalls the entire crawl, not just that request.
The default queues are LIFO (depth-first order), which keeps memory low for typical crawls. Requests are scheduled, not executed in yield order, so ordering between pages is never guaranteed.
Signals (spider_opened, item_scraped, request_dropped, spider_closed) and the stats collector expose these events, and extensions subscribe to them; that's how built-ins like CloseSpider, AutoThrottle and the memory usage monitor are implemented.
Example
import scrapy
from scrapy import signals
class FlowSpider(scrapy.Spider):
name = "flow"
start_urls = ["https://quotes.toscrape.com/"]
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
spider = super().from_crawler(crawler, *args, **kwargs)
crawler.signals.connect(spider.on_drop, signal=signals.request_dropped)
crawler.signals.connect(spider.on_closed, signal=signals.spider_closed)
return spider
def parse(self, response):
yield response.follow("/page/2/", priority=10) # jumps the queue
yield response.follow("/page/2/") # duplicate: dropped
yield response.follow("/", dont_filter=True, callback=self.parse_home)
def parse_home(self, response):
yield {"home_again": response.url}
def on_drop(self, request, spider):
self.logger.info("dropped %s", request.url)
def on_closed(self, spider, reason):
stats = self.crawler.stats.get_stats()
self.logger.info("filtered dupes: %s", stats.get("dupefilter/filtered"))Remember
- Flow: spider -> scheduler -> downloader (via middlewares) -> spider -> pipelines
- The dupefilter drops repeated request fingerprints; priority reorders
- Single-threaded event loop: blocking code stalls the whole crawl
- Signals and stats expose every step to extensions
Watch out for
- Calling a blocking HTTP client or time.sleep inside a callback, freezing every concurrent request.