TopolearnModern Scrapy code awaits async database drivers and APIs directly, which requires understanding how the asyncio reactor fits in.
Scrapy supports coroutines in many places: callbacks (async def parse(self, response), using yield to make an async generator), the spider's async def start(), item pipelines' process_item, and downloader middleware methods.
Inside them you can await:
- asyncio libraries (an async database driver, an async HTTP client for a side call), because Scrapy runs Twisted on the asyncio reactor (
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"), the default in current versions. - Twisted Deferreds, converted with
scrapy.utils.defer.maybe_deferred_to_future(d)so they're awaitable under asyncio.
An async callback that yields lets you do things like enrich an item from an API before yielding it, without blocking other requests: while one coroutine awaits, the event loop serves others.
Caveats:
- Awaiting inside a callback delays that callback's output; it doesn't add concurrency by itself. For many independent fetches, yield more Scrapy requests instead of awaiting in a loop.
- The reactor must be installed before anything imports
twisted.internet.reactor; installing it late raises an error about the wrong reactor. - CPU-heavy work still blocks the loop; offload it with
asyncio.to_threador a process pool.
Prefer Scrapy's own requests for crawling (they get throttling, retries, middlewares and stats); use direct awaits for auxiliary calls like databases or caches.
Example
import asyncio
import scrapy
async def lookup_country(author): # stand-in for an async DB/API call
await asyncio.sleep(0.05)
return {"Albert Einstein": "DE", "Jane Austen": "GB"}.get(author)
class AsyncQuotesSpider(scrapy.Spider):
name = "async_quotes"
start_urls = ["https://quotes.toscrape.com/"]
async def parse(self, response):
authors = response.css("small.author::text").getall()
countries = await asyncio.gather(*(lookup_country(a) for a in authors))
for author, country in zip(authors, countries):
yield {"author": author, "country": country}
class SaveToDbPipeline:
async def process_item(self, item):
await asyncio.sleep(0.01) # e.g. await db.insert(item)
return item
# TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor" (default)Remember
- Callbacks, start(), pipelines and middlewares can be coroutines
- The asyncio reactor lets you await asyncio libraries
- maybe_deferred_to_future makes Twisted Deferreds awaitable
- Awaiting doesn't add crawl concurrency: yield Requests for that
Watch out for
- Fetching hundreds of pages with an async HTTP client inside one callback, bypassing throttling, retries and stats.