TopolearnCrawlers in production run unattended on schedules, must survive restarts and report what happened.
Besides scrapy crawl, you can start crawls from Python. `AsyncCrawlerProcess` (or the older-style CrawlerProcess, whose methods return Twisted Deferreds instead of coroutines) starts the reactor, runs one or more spiders and blocks until they finish: process = AsyncCrawlerProcess(settings); process.crawl(MySpider, category="books"); process.start(). Keyword arguments become spider attributes (the same as -a category=books on the command line). Use get_project_settings() to load the project's settings.py. If your application already runs an event loop, AsyncCrawlerRunner / CrawlerRunner give you control without starting the reactor for you.
The reactor can't be restarted in the same process, so to run crawls repeatedly (in a scheduler, say), start a fresh process per crawl, for example via subprocess or a job runner.
Pause and resume: scrapy crawl big -s JOBDIR=crawls/big-1 persists the scheduler queue and seen-request fingerprints to disk. Stop with a single Ctrl-C (graceful shutdown) and rerun the same command to continue. Requests must be serializable for this, so callbacks must be spider methods.
Deployment options:
- Containers run by cron, Kubernetes CronJobs or Airflow tasks: simple and common.
- Scrapyd, a daemon with an HTTP API to deploy projects and schedule spiders.
- Hosted platforms such as Zyte Scrapy Cloud.
In all cases, ship logs and the final stats (item counts, error counts, finish reason) to your monitoring.
Example
# run_crawl.py
import scrapy
from scrapy.crawler import AsyncCrawlerProcess
class TitleSpider(scrapy.Spider):
name = "titles"
def __init__(self, start="https://quotes.toscrape.com/", **kw):
super().__init__(**kw)
self.start_urls = [start]
def parse(self, response):
yield {"url": response.url, "title": response.css("title::text").get()}
process = AsyncCrawlerProcess(settings={
"FEEDS": {"titles.jsonl": {"format": "jsonlines", "overwrite": True}},
"LOG_LEVEL": "INFO",
# "JOBDIR": "crawls/titles-1", # enable pause/resume
})
crawler = process.create_crawler(TitleSpider)
process.crawl(crawler, start="https://quotes.toscrape.com/")
process.start() # blocks until the crawl finishes
print(crawler.stats.get_value("item_scraped_count")) # 1
print(crawler.stats.get_value("finish_reason")) # finishedRemember
- AsyncCrawlerProcess (or CrawlerProcess) runs spiders from a script and blocks
- Keyword arguments to crawl() become spider arguments
- The reactor can't restart: one process per crawl run
- JOBDIR persists the queue and seen URLs for pause/resume
Watch out for
- Calling process.start() twice in one process and hitting ReactorNotRestartable.