TopolearnNetworks fail, servers return 500s and pages vanish, and a crawler must cope without silently losing data.
By default, responses with non-2xx status codes are dropped before reaching your callback by the HttpError spider middleware. To receive them anyway, set handle_httpstatus_list = [404] on the spider, meta={"handle_httpstatus_list": [404]} on a request, or HTTPERROR_ALLOWED_CODES globally.
The RetryMiddleware retries connection errors, timeouts and the codes in RETRY_HTTP_CODES (500, 502, 503, 504, 522, 524, 408, 429) up to RETRY_TIMES (default 2) extra times. Adjust per request with meta={"max_retry_times": 5} or disable with dont_retry. To retry from your own code (an error page served with status 200), use get_retry_request(response.request, spider=self, reason="empty page"), which respects the same limits and stats.
An errback is a function called when a request fails for good: scrapy.Request(url, callback=..., errback=self.on_error). It receives a Twisted Failure:
failure.check(HttpError),DNSLookupError,TimeoutErroridentify the cause.failure.requestis the failed request; for HttpError,failure.value.responseholds the response.
Use errbacks to log, record failed URLs as items for later reprocessing, or yield alternative requests.
At the end of a crawl, check the stats (downloader/response_status_count/404, retry/max_reached, log_count/ERROR). A crawl that "finished" with thousands of errors is a failure; alert on these numbers.
Example
import scrapy
from scrapy.downloadermiddlewares.retry import get_retry_request
from scrapy.spidermiddlewares.httperror import HttpError
from twisted.internet.error import DNSLookupError, TimeoutError
class RobustSpider(scrapy.Spider):
name = "robust"
handle_httpstatus_list = [404]
custom_settings = {"RETRY_TIMES": 3}
async def start(self):
for url in ["https://quotes.toscrape.com/", "https://quotes.toscrape.com/nope"]:
yield scrapy.Request(url, callback=self.parse, errback=self.on_error)
def parse(self, response):
if response.status == 404:
yield {"url": response.url, "status": "missing"}; return
if not response.css("div.quote"):
retry = get_retry_request(response.request, spider=self, reason="empty")
if retry: yield retry
return
yield {"url": response.url, "quotes": len(response.css("div.quote"))}
def on_error(self, failure):
if failure.check(HttpError):
self.logger.error("HTTP %s on %s", failure.value.response.status, failure.request.url)
elif failure.check(DNSLookupError, TimeoutError):
self.logger.error("Network error on %s", failure.request.url)Remember
- Non-2xx responses are filtered unless allowed via handle_httpstatus_list
- RetryMiddleware retries network errors and 5xx/429; tune RETRY_TIMES
- get_retry_request retries from callbacks with the same limits
- errback receives a Failure; check its type and log or record it
Watch out for
- Expecting a 404 page in the callback without handle_httpstatus_list, and never seeing it.