TopolearnA scraper is a program that fetches pages the way a browser does, so you need to know what that exchange looks like before Scrapy automates it.
When you type an address into your browser (Chrome, Firefox, Safari), it fetches the page from another computer and draws it for you. A scraper is a program that does the fetching part itself, then pulls the data it wants out of the page. That job is called web scraping.
The address is a URL, like https://books.toscrape.com/catalogue/page-2.html. It names the website (books.toscrape.com) and the page on it (/catalogue/page-2.html). Some URLs end with extra settings after a ?, like ?page=2.
The computer the website lives on is a server. Your browser or program is the client. They talk using HTTP, a simple rule book for asking and answering. The client sends a request: "please GET me /catalogue/page-2.html". The server sends back a response: the page, plus some information about it.
Every request has a method. GET means "send me this"; it's what loading a page uses. POST means "here's some data", and it's what a login or search form sends when you press its button.
Every response starts with a status code, a three-digit number saying how it went:
200: OK, here's the page.301/302: moved; go to this other URL instead. Following it is called a redirect.404: not found.403: forbidden.429: too many requests; slow down.500–504: something broke on the server.
Requests and responses also carry headers: short labelled notes such as Content-Type: text/html (what kind of data this is) or User-Agent: ... (which program is asking). Servers can also hand the client cookies, small values the client stores and sends back with later requests, for example to stay logged in.
A browser sends dozens of requests per page (images, styles, data). A scraper usually needs only the page itself, or the data behind it. Scrapy automates all of this: it sends the requests, waits for the responses, and gives them to your code.
Example
# What one exchange looks like, written out as text.
# The request (client -> server):
GET /catalogue/page-2.html HTTP/1.1
Host: books.toscrape.com
User-Agent: bookbot/1.0 (+mailto:me@example.com)
# The response (server -> client):
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Set-Cookie: session=abc123
<html> ... the page itself ... </html>Remember
- A URL names a site and a page on it
- The client sends a request (GET or POST); the server sends back a response
- The status code says how it went: 200 OK, 3xx moved, 4xx your problem, 5xx theirs
- Headers carry extra details; cookies let a site recognise you on later requests
Watch out for
- Treating any response as success; a 404 or 500 error still comes back with a page in it.