TopolearnMany crawls are 'follow these kinds of links, parse those kinds of pages', which rules express in a few lines.
CrawlSpider is a Spider subclass driven by rules. Each Rule wraps a `LinkExtractor` that finds links in every response, plus what to do with them:
LinkExtractor(allow=r"/catalogue/category/"): regexes the URL must match;deny=excludes;restrict_css=/restrict_xpaths=limit extraction to a page region such as the main content or pagination.Rule(extractor, callback="parse_book"): parse the linked pages with that method.follow=True(default when there's no callback): keep applying the rules to those pages too.
Rules are applied in order, and a link is handled by the first rule that matches it, so put specific rules before general ones.
A typical setup has one rule that follows category and pagination links without a callback, and one rule that sends product pages to parse_item.
The crucial gotcha: don't define a method called `parse` in a CrawlSpider. CrawlSpider uses parse internally to apply the rules; overriding it silently disables the crawling. Name callbacks parse_item or similar. To also parse the start pages, override parse_start_url.
LinkExtractor is useful outside CrawlSpider too: LinkExtractor(...).extract_links(response) returns Link objects with .url and .text, deduplicated and absolute.
For sites that publish one, SitemapSpider crawls from sitemap.xml instead, often the most efficient way to reach every page.
Example
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
class BooksSpider(CrawlSpider):
name = "books"
allowed_domains = ["books.toscrape.com"]
start_urls = ["https://books.toscrape.com/"]
rules = (
# book detail pages -> parse_book (and don't follow further)
Rule(LinkExtractor(restrict_css="article.product_pod h3"),
callback="parse_book", follow=False),
# category and pagination pages -> just follow
Rule(LinkExtractor(restrict_css=["ul.nav-list", "li.next"])),
)
def parse_book(self, response): # never name this "parse"
yield {
"title": response.css("div.product_main h1::text").get(),
"price": response.css("p.price_color::text").get(),
"url": response.url,
}Remember
- CrawlSpider + Rule(LinkExtractor(...)) crawls declaratively
- allow/deny regexes and restrict_css focus link extraction
- The first matching rule handles a link: order matters
- Never override parse in a CrawlSpider
Watch out for
- Naming a CrawlSpider callback parse and wondering why no links are followed.