TopolearnLogins, search forms and the JSON endpoints behind modern pages are where most valuable data hides.
Beyond response.follow, you can construct requests explicitly: scrapy.Request(url, method="POST", headers={...}, body=..., callback=..., cb_kwargs=..., meta=...). To post form-encoded data you already know, FormRequest(url, formdata={...}) url-encodes it for you.
HTML forms: logins and search pages usually carry hidden fields (CSRF tokens, view state) you must send back. Recent Scrapy versions recommend the form2request library for this (pip install form2request): form2request(response.css("form#login"), {"username": "u", "password": "p"}) reads the form, keeps its hidden inputs, applies your values, and resolves the action URL and method; .to_scrapy(callback=...) turns the result into a Scrapy request. Older code uses FormRequest.from_response(response, formdata=...), which does the same but is deprecated in current releases.
Cookies are handled automatically by the cookies middleware, so after logging in, later requests in the same crawl stay logged in.
JSON APIs: many sites load data from JSON endpoints visible in the browser's network tab. Call them directly:
response.json()parses a JSON response body.scrapy.http.JsonRequest(url, data={...})sends a JSON body with the right headers (POST by default when data is given).- Paginate by computing the next page from the JSON (
page + 1, a cursor field) and yielding a new request until it's empty.
APIs are usually faster, more stable and lighter on the server than HTML pages. Still respect the site's terms, authentication and rate limits.
Example
import scrapy
from form2request import form2request
class AccountSpider(scrapy.Spider):
name = "account"
start_urls = ["https://quotes.toscrape.com/login"]
def parse(self, response):
# keeps hidden fields like csrf_token, fills in ours
req = form2request(response.css("form"), {"username": "demo", "password": "demo"})
yield req.to_scrapy(callback=self.after_login)
def after_login(self, response):
if "Logout" not in response.text:
self.logger.error("login failed"); return
yield response.follow("/api/quotes?page=1", callback=self.parse_api)
def parse_api(self, response):
data = response.json()
for q in data["quotes"]:
yield {"author": q["author"]["name"], "text": q["text"]}
if data["has_next"]:
yield response.follow(f"/api/quotes?page={data['page'] + 1}",
callback=self.parse_api)Remember
- scrapy.Request takes method, headers, body, callback, cb_kwargs
- form2request builds a submission from a real form, keeping hidden fields
- Cookies persist across requests automatically
- response.json() and JsonRequest for JSON APIs
Watch out for
- Posting login data with a plain FormRequest and missing the page's CSRF token.