TopolearnDeveloping selectors interactively is far faster than rerunning a whole crawl after every tweak.
scrapy shell "https://quotes.toscrape.com/" downloads the page with Scrapy's own downloader (same settings, headers and middlewares as your project when run inside it) and opens a Python shell (IPython if installed) with objects ready:
response: the downloaded page; tryresponse.css(...)andresponse.xpath(...)right away.request,settings,spider,crawler.fetch(url_or_request): download another page and replaceresponse.view(response): open the response in your browser as Scrapy received it, which is the fastest way to see whether content is missing because JavaScript renders it.shelp(): list the available objects and shortcuts.
You can also run the shell on a local file (scrapy shell ./page.html) to work offline with a saved page.
Inside a running spider, from scrapy.shell import inspect_response and calling inspect_response(response, self) in a callback drops you into a shell at exactly that point, with the live response, great for debugging pages that behave differently from what you expected.
A related command, scrapy fetch --nolog URL, prints the raw body as Scrapy sees it, and scrapy view URL opens it in a browser.
Workflow: explore in the shell, copy the selector into the spider, then run the spider on a few pages with -s CLOSESPIDER_ITEMCOUNT=5 to check output.
Example
# $ scrapy shell "https://quotes.toscrape.com/"
>>> response.status
200
>>> response.css("title::text").get()
'Quotes to Scrape'
>>> q = response.css("div.quote")[0]
>>> q.css("small.author::text").get()
'Albert Einstein'
>>> q.css("a.tag::text").getall()
['change', 'deep-thoughts', 'thinking', 'world']
>>> fetch("https://quotes.toscrape.com/page/2/")
>>> response.url
'https://quotes.toscrape.com/page/2/'
>>> view(response) # opens the downloaded HTML in your browser
TrueRemember
- scrapy shell URL gives you response, fetch() and view()
- view(response) shows the page as Scrapy got it, without JavaScript
- inspect_response(response, self) opens a shell inside a callback
- Develop selectors in the shell, then paste them into the spider
Watch out for
- Testing selectors in the browser's dev tools (which show the JavaScript-rendered DOM) instead of the raw response.