Settings and polite crawling

Crawling too aggressively gets you blocked and can hurt the sites you depend on; Scrapy has the knobs built in.

Settings come from several levels, lowest to highest priority: Scrapy defaults, project settings.py, the spider's custom_settings dict, and the command line (-s NAME=value).

Politeness and identity:

ROBOTSTXT_OBEY = True: respect robots.txt (the project template enables it; the global default is False).
USER_AGENT: identify your bot, ideally with a contact URL.
DOWNLOAD_DELAY = 1: seconds between requests to the same site, randomized between 0.5x and 1.5x by default (DOWNLOAD_DELAY_JITTER = 0.5; older versions used RANDOMIZE_DOWNLOAD_DELAY).
CONCURRENT_REQUESTS (default 16 overall) and CONCURRENT_REQUESTS_PER_DOMAIN (default 8; the project template sets 1).

AutoThrottle (AUTOTHROTTLE_ENABLED = True) adapts the delay to the server's response latency, aiming at AUTOTHROTTLE_TARGET_CONCURRENCY parallel requests per server. It slows down automatically when the site does, which is usually better than a fixed guess.

Development helpers:

HTTPCACHE_ENABLED = True caches responses on disk, so reruns while you tweak parsing don't hit the site.
CLOSESPIDER_ITEMCOUNT / CLOSESPIDER_PAGECOUNT stop early.
LOG_LEVEL = "INFO" cuts noise.

Read settings in code via self.settings.getint("DOWNLOAD_DELAY") in a spider or crawler.settings in components. scrapy settings --get NAME shows the effective value inside a project.

Example

# settings.py
BOT_NAME = "bookstore"
USER_AGENT = "bookstore-bot/1.0 (+https://example.com/bot)"
ROBOTSTXT_OBEY = True

CONCURRENT_REQUESTS_PER_DOMAIN = 2
DOWNLOAD_DELAY = 0.5

AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 30
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0

HTTPCACHE_ENABLED = True          # dev only: reuse downloaded pages
HTTPCACHE_EXPIRATION_SECS = 3600
FEED_EXPORT_ENCODING = "utf-8"

# One-off overrides from the command line:
# $ scrapy crawl books -s LOG_LEVEL=INFO -s CLOSESPIDER_ITEMCOUNT=20

Remember

Priority: defaults < settings.py < custom_settings < -s on the command line
Obey robots.txt, identify yourself, and limit per-domain concurrency
AutoThrottle adapts delays to server latency
HTTPCACHE_ENABLED speeds up development reruns

Watch out for

Leaving HTTPCACHE_ENABLED on in production and scraping stale pages.

Check yourself

A spider sets custom_settings = {'DOWNLOAD_DELAY': 2} and you run with -s DOWNLOAD_DELAY=0.5. Which delay applies?

Official docs: docs.scrapy.org/en/latest/topics/settings.html

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks