TopolearnScrapers break constantly; the design choices here decide whether a break is a crash, silent bad data or an easy fix.
Pages change without notice. A few habits make scrapers resilient:
- Anchor on meaning, not position: ids, data attributes, semantic classes, label text ("Price") and structured data, not
div:nth-child(4). - Use structured data when present: many pages embed JSON-LD in
<script type="application/ld+json">or Open Graph<meta property="og:...">tags. Parsing that JSON is far more stable than scraping visible markup. - Isolate field extraction in small functions that return
Nonefor missing data, and validate records afterwards: a required field going missing across all records means the layout changed, so fail loudly. - Save raw HTML of fetched pages so you can re-parse after fixing a bug without refetching.
Dynamic pages: if the data you see in the browser isn't in resp.content, JavaScript loads it. Check "View source" (not the inspector, which shows the live DOM). Options, from best to worst:
- Find the underlying JSON API in the browser's network tab and call it directly.
- Read the initial state embedded in a script tag (
window.__INITIAL_STATE__ = {...}). - Render with a headless browser (Playwright) and pass
page.content()to Beautiful Soup.
For large crawls with many pages, retries and pipelines, a framework like Scrapy is a better fit than a loop around requests.
Example
import json
from bs4 import BeautifulSoup
html = """<html><head>
<meta property="og:title" content="Steel Kettle">
<script type="application/ld+json">
{"@type": "Product", "name": "Steel Kettle", "offers": {"price": "29.90", "priceCurrency": "EUR"}}
</script></head><body><h1 class="pdp-title-x7">Steel Kettle</h1></body></html>"""
soup = BeautifulSoup(html, "html.parser")
def product_from_jsonld(soup):
for s in soup.find_all("script", type="application/ld+json"):
data = json.loads(s.string or "{}")
if data.get("@type") == "Product":
return {"name": data["name"], "price": float(data["offers"]["price"])}
return None
def og(soup, prop):
tag = soup.find("meta", property=f"og:{prop}")
return tag["content"] if tag else None
print(product_from_jsonld(soup)) # {'name': 'Steel Kettle', 'price': 29.9}
print(og(soup, "title"), og(soup, "image")) # Steel Kettle NoneRemember
- Prefer ids, data attributes, labels and structured data over positions
- JSON-LD and Open Graph tags are stable data sources
- Validate records and fail loudly when required fields vanish
- Content missing from the raw HTML is rendered by JavaScript: find the API or render
Watch out for
- Scraping a generated class like pdp-title-x7 that changes with every frontend build.