TopolearnSelectors are how every value gets out of a page, and knowing both CSS and XPath lets you handle any structure.
A response offers response.css(query) and response.xpath(query). Both return a SelectorList, which you can query further or extract from:
.get()returns the first match as a string, orNone(orget(default=""))..getall()returns a list of all matches..re(r"pattern")/.re_first(...)apply a regex to the matches.
CSS in Scrapy has two extensions (from the parsel library) to select text and attributes:
span.text::textselects the text nodes of the element.a::attr(href)selects an attribute's value. Alternativelyresponse.css("a").attrib["href"]reads an attribute of the first match.
XPath is more powerful for some tasks: //a[contains(text(), "Next")]/@href, //th[text()="Price"]/following-sibling::td/text(), or string(.) for all text inside an element. CSS queries are actually translated to XPath under the hood, and you can chain both: response.css("div.product").xpath(".//h2/text()").
When looping over elements, query relative to each one: quote.css("span.text::text") stays inside that quote. In XPath, start relative expressions with .// instead of //, which would search the whole document again.
::text only returns direct text nodes; for text spread over nested tags use *::text with getall() and join, or XPath string().
Example
from scrapy.http import HtmlResponse
body = b"""<div class="product" data-sku="A1"><h2>Kettle <small>steel</small></h2>
<table><tr><th>Price</th><td>29.90 EUR</td></tr></table>
<a class="next" href="/page/2">Next page</a></div>"""
response = HtmlResponse(url="https://shop.example/", body=body, encoding="utf-8")
p = response.css("div.product")
print(p.attrib["data-sku"]) # A1
print(p.css("h2::text").get()) # 'Kettle '
print("".join(p.css("h2 *::text, h2::text").getall()).strip()) # Kettle steel
print(p.xpath("string(.//h2)").get()) # Kettle steel
print(p.xpath('.//th[text()="Price"]/following-sibling::td/text()').re_first(r"[\d.]+")) # 29.90
print(response.css("a.next::attr(href)").get()) # /page/2
print(response.css("span.missing::text").get(default="n/a")) # n/aRemember
- get() for the first match, getall() for all, re()/re_first() for regex
- ::text and ::attr(name) are Scrapy's CSS extensions
- Query relative to each element; in XPath use .// not //
- XPath handles text conditions and siblings; string() joins nested text
Watch out for
- Using // inside a loop over elements and getting matches from the whole page every time.