TopolearnReal crawls span listing pages, detail pages and pagination, and Scrapy's request model makes that natural.
To crawl further, a callback yields requests. The convenient way is response.follow(url, callback=...):
- It accepts relative URLs (
/page/2/) and resolves them against the response URL. - It also accepts a selector for an
<a>element directly:response.follow(a_selector, ...). response.follow_all(css="li.next a", callback=self.parse)creates requests for every match.
The callback defaults to self.parse. Use different callbacks for different page types: parse for listing pages yields requests to detail pages with callback=self.parse_author, and a request for the next listing page with callback=self.parse.
Deduplication: Scrapy remembers request fingerprints and drops requests for URLs it has already seen, so following the same author link from many quotes fetches it once. Pass dont_filter=True to bypass that when needed.
To pass data from one page to the next, use cb_kwargs={"category": name}: the values arrive as keyword arguments of the callback. That's the clean replacement for stuffing data into request.meta.
Pagination ends naturally when there's no next link. Guard deep or infinite crawls with DEPTH_LIMIT or CLOSESPIDER_PAGECOUNT, and keep allowed_domains set so links to other sites are ignored.
Example
import scrapy
class AuthorsSpider(scrapy.Spider):
name = "authors"
allowed_domains = ["quotes.toscrape.com"]
start_urls = ["https://quotes.toscrape.com/"]
def parse(self, response):
for q in response.css("div.quote"):
about = q.css("span a") # the (about) link
yield response.follow(about[0], callback=self.parse_author,
cb_kwargs={"first_quote": q.css("span.text::text").get()})
yield from response.follow_all(css="li.next a", callback=self.parse)
def parse_author(self, response, first_quote):
yield {
"name": response.css("h3.author-title::text").get().strip(),
"born": response.css("span.author-born-date::text").get(),
"first_quote_seen": first_quote[:30],
}
# Each author page is fetched once, even if linked from many quotes.Remember
- response.follow resolves relative URLs and accepts <a> selectors
- Use separate callbacks per page type
- Duplicate URLs are filtered automatically (dont_filter=True to override)
- cb_kwargs passes data to the next callback
Watch out for
- Building absolute URLs by string concatenation instead of response.follow / response.urljoin.