Yielding items and exporting feeds

Getting clean output files without writing any export code is one of Scrapy's biggest time savers.

Whatever a callback yields that isn't a request is treated as an item: a dict, a Scrapy Item, a dataclass or an attrs object. Items flow through the item pipelines and then to the configured feed exports.

From the command line:

scrapy crawl quotes -O quotes.json writes (overwriting) a JSON array.
-o quotes.jsonl appends; use JSON Lines (.jsonl) for appending, since appending to a JSON array produces an invalid file.
.csv and .xml work too; the format follows the extension, or force it with -o out.txt:jsonlines.

In code or settings, the FEEDS setting configures one or more outputs with options:

FEEDS = {"data/%(name)s-%(time)s.jsonl": {"format": "jsonlines", "encoding": "utf8"}} uses placeholders for the spider name and time.
"fields" selects and orders CSV columns; "overwrite": True replaces existing files.
URIs can point to S3 (s3://bucket/key), FTP or Google Cloud Storage with the right extras installed.

Set FEED_EXPORT_ENCODING = "utf-8" (the generated project template already does) so non-ASCII text isn't written as \u00e9 escapes in JSON.

Yield items as soon as they're extracted rather than collecting lists: memory stays flat and results stream out even if the crawl stops halfway.

Example

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]
    custom_settings = {
        "FEEDS": {
            "out/%(name)s.jsonl": {"format": "jsonlines", "overwrite": True},
            "out/%(name)s.csv": {"format": "csv", "fields": ["author", "text"]},
        },
        "FEED_EXPORT_ENCODING": "utf-8",
    }

    def parse(self, response):
        for q in response.css("div.quote"):
            yield {
                "text": q.css("span.text::text").get(),
                "author": q.css("small.author::text").get(),
                "tags": q.css("a.tag::text").getall(),
            }

# $ scrapy runspider quotes.py            -> out/quotes.jsonl and out/quotes.csv
# $ scrapy runspider quotes.py -O q.json  -> also a JSON array

Remember

Anything yielded that isn't a Request is an item
-O overwrites, -o appends; append only to .jsonl
FEEDS configures formats, fields, paths and remote storage
Set FEED_EXPORT_ENCODING = 'utf-8' for readable JSON

Watch out for

Appending with -o to an existing .json file, producing two concatenated JSON arrays.

Check yourself

Which output is safe to append to across multiple runs?

Official docs: docs.scrapy.org/en/latest/topics/feed-exports.html

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks