TopolearnDeclared item fields catch typos and document your data, and loaders keep cleaning logic out of callbacks.
Plain dicts work, but larger projects declare item types. Scrapy supports, through the itemadapter library:
scrapy.Itemsubclasses withscrapy.Field()attributes: assigning an undeclared field raisesKeyError, which catches typos.- dataclasses and attrs classes, giving type hints and defaults.
Pipelines and exporters handle all of them uniformly via ItemAdapter(item).
ItemLoader separates where data comes from and how it's cleaned:
loader = ItemLoader(item=Product(), response=response)(orselector=for a sub-element).loader.add_css("name", "h1::text"),add_xpath(...),add_value("url", response.url); each call can add several values to a field.loader.load_item()returns the populated item.
Every value passes through processors from the itemloaders.processors module:
- Input processors run on each extracted value as it's added:
MapCompose(str.strip, to_float)applies functions in order and dropsNoneresults. - Output processors run once on the collected list when loading:
TakeFirst()returns the first non-empty value,Join(", ")concatenates.
Configure them per field (name_in, price_out attributes on a loader subclass, or Field(input_processor=...)) and set default_output_processor = TakeFirst(), since otherwise every field is a list.
Loaders shine on sites with many similar pages; for a quick spider, plain dicts with careful selectors are fine.
Example
from dataclasses import dataclass
from itemloaders.processors import MapCompose, TakeFirst, Join
from scrapy.http import HtmlResponse
from scrapy.loader import ItemLoader
@dataclass
class Product:
name: str | None = None
price: float | None = None
tags: str | None = None
class ProductLoader(ItemLoader):
default_item_class = Product
default_output_processor = TakeFirst()
name_in = MapCompose(str.strip)
price_in = MapCompose(lambda s: s.replace("EUR", "").strip(), float)
tags_out = Join(", ")
body = b'<h1> Kettle </h1><span class="price">29.90 EUR</span><a class="tag">home</a><a class="tag">steel</a>'
response = HtmlResponse(url="https://shop.example/p/1", body=body, encoding="utf-8")
loader = ProductLoader(response=response)
loader.add_css("name", "h1::text")
loader.add_css("price", "span.price::text")
loader.add_css("tags", "a.tag::text")
print(loader.load_item())
# Product(name='Kettle', price=29.9, tags='home, steel')Remember
- scrapy.Item, dataclasses and attrs all work as items via ItemAdapter
- ItemLoader: add_css / add_xpath / add_value, then load_item()
- Input processors clean each value; output processors combine the list
- Set default_output_processor = TakeFirst() to avoid list fields
Watch out for
- Forgetting an output processor and exporting every field as a one-element list.