Topolearn

Learn Scrapy

Python framework for fast, polite, large-scale web crawling and scraping

17 lessons · 3 hours · 4 levels · covers Scrapy 2.19

Scrapy is a Python framework for crawling websites and extracting structured data. You write spiders that yield requests and items, and Scrapy handles scheduling, concurrency, retries, throttling, deduplication, pipelines and export. It's the standard choice when scraping goes beyond a few pages: whole sites, recurring crawls and production data pipelines.

Sign in free to track progressOfficial docs ↗

Before you start

Scrapy builds on these; the tree links to the exact lessons you need.

PythonPractical level

How deep do you want to go?

1. EssentialsRead and write simple code, do the everyday basics5 lessons · 54 min in total
2. PracticalUse it productively at work, with the usual idioms10 lessons · 2 hours in total
3. AdvancedThe less common features, performance and trade-offs14 lessons · 3 hours in total
4. ExpertInternals and the edge cases experts know17 lessons · 3 hours in total

The Scrapy tree

ScrapyProjects and your first spiderProjects and yourfirst spiderSelectors: CSS and XPathSelectors: CSS andXPathYielding items and exporting feedsYielding items andexporting feedsFollowing links and paginationFollowing links andpaginationExploring pages in scrapy shellExploring pages inscrapy shellItems and ItemLoadersItems andItemLoadersItem pipelinesItem pipelinesSettings and polite crawlingSettings and politecrawlingCrawlSpider and link extraction rulesCrawlSpider and linkextraction rulesRequests, forms and JSON APIsRequests, forms andJSON APIsClasses and instancesClasses andinstancesIterators and generatorsIterators andgeneratorsDataclassesDataclassesasync and await with asyncioasync and await withasyncio
Essentials and Practical shown (10 of 17 lessons); the full list is below. Click a lesson to read it.

All 17 lessons, in order

Getting started

  1. Projects and your first spider · Essentials · 12 min

    Create a Scrapy project, write a Spider with start_urls and a parse callback, and run it with scrapy crawl.

Extracting data

  1. Selectors: CSS and XPath · Essentials · 12 min

    Extract data from responses with response.css() and response.xpath(), using ::text, ::attr(), get() and getall().

  2. Yielding items and exporting feeds · Essentials · 10 min

    Yield dict items from callbacks and export them to JSON, JSON Lines or CSV with -O/-o or the FEEDS setting.

  3. Exploring pages in scrapy shell · Essentials · 8 min

    Use scrapy shell to fetch a page and try selectors interactively before putting them in a spider.

Crawling

  1. Following links and pagination · Essentials · 12 min

    Crawl beyond the first page with response.follow and follow_all, separate callbacks per page type, and pass data with cb_kwargs.

  2. Settings and polite crawling · Practical · 10 min

    Configure robots.txt obedience, delays, concurrency, AutoThrottle, user agent and HTTP cache through Scrapy settings.

  3. CrawlSpider and link extraction rules · Practical · 10 min

    Crawl whole sections of a site declaratively with CrawlSpider, Rule and LinkExtractor patterns.

  4. Requests, forms and JSON APIs · Practical · 12 min

    Build custom Requests with headers and methods, submit HTML forms with form2request, and consume JSON APIs.

Items and pipelines

  1. Items and ItemLoaders · Practical · 12 min

    Define item schemas with scrapy.Item or dataclasses, and populate them cleanly with ItemLoader input and output processors.

  2. Item pipelines · Practical · 12 min

    Validate, clean, deduplicate and store items in pipeline classes enabled and ordered through ITEM_PIPELINES.

Middlewares, errors and production

  1. Downloader and spider middlewares · Advanced · 14 min

    Hook into every request and response with downloader middlewares (headers, proxies, custom responses) and know where spider middlewares fit.

  2. Errors, retries and errbacks · Advanced · 12 min

    Handle failed requests with errbacks, control retries and HTTP error filtering, and keep partial failures visible.

  3. JavaScript-rendered pages · Advanced · 12 min

    Handle pages whose content is rendered by JavaScript: find the underlying API, parse embedded JSON, or render with scrapy-playwright.

  4. Running from scripts and in production · Advanced · 12 min

    Run crawls from Python with CrawlerProcess, persist and resume long crawls with JOBDIR, and deploy spiders to schedulers or Scrapyd.

Architecture and scale

  1. Architecture: engine, scheduler and data flow · Expert · 14 min

    How the engine, scheduler, downloader, middlewares, spiders and pipelines interact, and how request deduplication and priorities work.

  2. async callbacks and asyncio integration · Expert · 12 min

    Write async def callbacks and pipelines, await asyncio libraries under the asyncio reactor, and bridge Twisted Deferreds.

  3. Broad crawls and scaling out · Expert · 12 min

    Tune Scrapy for crawling many domains: concurrency, BFO ordering, DownloaderAwarePriorityQueue, memory limits and distributed crawling.