TopolearnPipelines keep storage and validation out of spiders, so the same logic applies to every spider in a project.
After a callback yields an item, it passes through the item pipelines: classes with a process_item(self, item) method, enabled in settings:
ITEM_PIPELINES = {"bookstore.pipelines.ValidatePipeline": 100, "bookstore.pipelines.PostgresPipeline": 800}
The numbers (0 to 1000) set the order: lower runs first. Each process_item must either return the item (possibly modified) to pass it on, or raise `DropItem` to discard it, which is logged and counted in the stats.
Typical pipelines, in order:
- Validation: drop items missing required fields.
- Cleaning and normalization: prices to numbers, dates to ISO format.
- Deduplication: keep a set of seen keys and drop repeats.
- Storage: write to a database or API.
Lifecycle hooks: open_spider() and close_spider() run once per crawl, the place to open and close database connections. A from_crawler(cls, crawler) classmethod gives access to settings (connection strings) and the crawler.
Use ItemAdapter(item) to read and write fields so the pipeline works with dicts, Items and dataclasses alike.
process_item may also be an async def coroutine, which suits async database drivers. Older Scrapy versions passed a spider argument to these methods; current versions deprecate requiring it in favor of from_crawler and crawler.spider.
Example
# bookstore/pipelines.py
from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem
class ValidatePipeline:
def process_item(self, item):
a = ItemAdapter(item)
if not a.get("title") or a.get("price") is None:
raise DropItem(f"missing fields: {a.asdict()}")
a["price"] = round(float(a["price"]), 2)
return item
class DedupePipeline:
def open_spider(self):
self.seen = set()
def process_item(self, item):
key = ItemAdapter(item)["title"].lower()
if key in self.seen:
raise DropItem(f"duplicate: {key}")
self.seen.add(key)
return item
# settings.py
ITEM_PIPELINES = {"bookstore.pipelines.ValidatePipeline": 100,
"bookstore.pipelines.DedupePipeline": 200}Remember
- process_item returns the item to pass it on or raises DropItem
- ITEM_PIPELINES numbers set the order: lower first
- open_spider / close_spider manage resources; from_crawler reads settings
- Use ItemAdapter to support every item type
Watch out for
- Forgetting to return the item, so every later pipeline and the feed receive None.