TopolearnBeautiful Soup is almost always paired with an HTTP client, and the glue code decides whether the scraper is correct and polite.
A minimal scraper: fetch with requests, check the status, parse, extract.
- Use a
requests.Session()to reuse connections and set headers once, including an honestUser-Agentwith contact info. - Always pass a
timeout=and callresp.raise_for_status()so errors don't parse as content. - Pass
resp.content(bytes) to Beautiful Soup and let it detect the encoding from the document;resp.textrelies on HTTP headers that are often wrong.
Links in pages are often relative (/p/2, ../img.png). Resolve them against the page URL with urllib.parse.urljoin(resp.url, href); use resp.url rather than the URL you requested, since redirects may have moved you.
Pagination usually means following a "next" link until there isn't one.
Be a good citizen:
- Check
robots.txt(the standard library'surllib.robotparserreads it) and the site's terms. - Rate-limit with a delay between requests and avoid parallel hammering.
- Prefer an official API or a JSON endpoint the page already calls (look in the browser's network tab), which is more stable than HTML.
- Cache pages while developing so you don't refetch on every run.
If the content isn't in the HTML at all because JavaScript renders it, Beautiful Soup can't see it; see the lesson on robust scrapers.
Example
import time
from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup
session = requests.Session()
session.headers["User-Agent"] = "price-bot/1.0 (+mailto:me@example.com)"
def scrape_titles(start_url, max_pages=3):
url, titles = start_url, []
for _ in range(max_pages):
resp = session.get(url, timeout=10)
resp.raise_for_status()
soup = BeautifulSoup(resp.content, "html.parser")
titles += [a.get_text(strip=True) for a in soup.select("h3 a")]
nxt = soup.select_one("li.next a")
if nxt is None:
break
url = urljoin(resp.url, nxt["href"]) # resolve relative link
time.sleep(1) # be polite
return titles
print(scrape_titles("https://books.toscrape.com/")[:2])
# ['A Light in the ...', 'Tipping the Velvet']Remember
- requests.Session with User-Agent, timeout and raise_for_status
- Pass resp.content (bytes) so Beautiful Soup detects the encoding
- urljoin(resp.url, href) for relative links
- Respect robots.txt, rate-limit, and prefer APIs when they exist
Watch out for
- Building next-page URLs by string concatenation and breaking on relative paths.