TopolearnMany modern sites ship empty HTML shells, and choosing the right technique decides between a fast crawler and a slow, fragile one.
Scrapy downloads HTML; it doesn't run JavaScript. If view(response) in the shell shows a page without the data, the content is rendered client-side. In order of preference:
- Find the data source: open the browser's network tab, filter by XHR/fetch, and look for JSON responses containing the data. Reproduce the request in Scrapy (same URL, query parameters and important headers) and parse with
response.json(). This is usually the fastest, most robust option. - Embedded state: frameworks often put initial data in the HTML, in
<script id="__NEXT_DATA__" type="application/json">,window.__INITIAL_STATE__ = {...}or JSON-LD. Extract the script text with a selector andjson.loadsit (a regex or the chompjs library helps with JavaScript object literals). - Render the page with a headless browser via scrapy-playwright: install it, configure its download handler and the asyncio reactor, and mark requests with
meta={"playwright": True}. You can wait for selectors, click and scroll via page methods. Rendering is orders of magnitude slower and heavier, so render only the pages that need it.
Hosted rendering and anti-bot services exist too (for example Zyte API), integrated as download handlers or middlewares.
Whatever you choose, keep extraction code separate from fetching, so switching from rendered HTML to a discovered API later doesn't require rewriting your parsers.
Example
import json
import scrapy
class ListingsSpider(scrapy.Spider):
name = "listings"
start_urls = ["https://example.com/listings"]
def parse(self, response):
# Option 1: data embedded by a Next.js app
raw = response.css("script#__NEXT_DATA__::text").get()
if raw:
for item in json.loads(raw)["props"]["pageProps"]["listings"]:
yield {"id": item["id"], "title": item["title"]}
return
# Option 2: render with a browser (scrapy-playwright)
yield scrapy.Request(response.url, callback=self.parse_rendered,
meta={"playwright": True}, dont_filter=True)
def parse_rendered(self, response):
for card in response.css("div.listing"):
yield {"title": card.css("h2::text").get()}
# settings.py for scrapy-playwright:
# DOWNLOAD_HANDLERS = {"https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
# "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler"}Remember
- Check view(response): missing data means client-side rendering
- Best: call the JSON API the page uses
- Next: parse embedded JSON state from script tags
- Last resort: render with scrapy-playwright, only for pages that need it
Watch out for
- Rendering every page with a browser when one JSON endpoint would return all the data.