Projects and your first spider

Scrapy's project layout and spider lifecycle are the frame every crawler you write will fit into.

Scrapy is a framework for crawling websites and extracting structured data. Unlike a requests-plus-parser script, it handles scheduling, concurrency, retries, throttling, deduplication and export for you; you write spiders that say what to fetch and how to parse it.

Install it with pip install scrapy, then:

scrapy startproject bookstore creates a project: settings.py, items.py, pipelines.py, middlewares.py and a spiders/ package.
scrapy genspider quotes quotes.toscrape.com adds a spider skeleton.
scrapy crawl quotes runs the spider by its name.
For a single file without a project, scrapy runspider quotes.py works too.

A spider is a class subclassing scrapy.Spider with:

name: unique id used by scrapy crawl.
allowed_domains: requests to other domains are dropped by the offsite middleware.
start_urls: the first pages to fetch. For custom start requests, define async def start(self) and yield scrapy.Request objects. Older code uses start_requests(), deprecated in 2.13 and since removed, so current versions never call it.
parse(self, response): the default callback, called with each downloaded response. It's a generator that yields extracted items (dicts) and new requests.

Everything is asynchronous: yielding a request doesn't download it immediately; Scrapy schedules it and calls your callback later, possibly interleaved with many other pages.

Example

# quotes.py   (run with: scrapy runspider quotes.py -O quotes.json)
import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    allowed_domains = ["quotes.toscrape.com"]
    start_urls = ["https://quotes.toscrape.com/"]

    def parse(self, response):
        self.logger.info("Visited %s (status %s)", response.url, response.status)
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
            }

# $ scrapy startproject bookstore && cd bookstore
# $ scrapy genspider quotes quotes.toscrape.com
# $ scrapy crawl quotes
# ... 'item_scraped_count': 10, 'response_received_count': 2, ...

Remember

startproject creates the layout; genspider adds a spider
A Spider has name, start_urls (or async start()) and parse callbacks
Callbacks yield items and new requests
Requests are scheduled and downloaded asynchronously

Watch out for

Returning a list once at the end of parse instead of yielding items as you go (works, but delays everything and holds memory).

Check yourself

What does `scrapy crawl quotes` use to find the spider?

Official docs: docs.scrapy.org/en/latest/intro/tutorial.html

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks