TopolearnAt thousands of pages, parsing time and memory dominate, and a few switches make a several-fold difference.
Beautiful Soup is written in Python on top of a parser, building a full object tree with every tag and string. That's convenient but costly. Ways to speed it up:
- Use lxml as the parser: typically several times faster than html.parser and far faster than html5lib.
- Parse only what you need with a
SoupStrainer:BeautifulSoup(html, "lxml", parse_only=SoupStrainer("a"))builds a tree of just the matching tags, skipping everything else. It accepts the same filters asfind_all. It doesn't work with html5lib, which always builds the whole tree. - Pass bytes with a known encoding (
from_encoding=) or install a fast charset detector, since encoding detection can be surprisingly expensive. - Search narrowly: find a container once, then search inside it, instead of repeated
find_allover the whole document.
Memory: every NavigableString you keep holds a reference into the tree, keeping the whole document alive. Convert results to plain str (str(tag.string) or get_text()) before storing them, and call soup.decompose() when done with large documents in long-running processes.
When parsing is still the bottleneck, use lxml's own API (lxml.html with XPath or CSS via cssselect), or a fast HTML5 parser such as selectolax, and keep Beautiful Soup for the messy cases. Measure with time.perf_counter() or a profiler before optimizing.
Example
import time
from bs4 import BeautifulSoup, SoupStrainer
html = "<html><body>" + "".join(
f'<div class="row"><p>Item {i}</p><a href="/i/{i}">more</a></div>' for i in range(5000)
) + "</body></html>"
def timed(label, fn):
t = time.perf_counter(); result = fn()
print(f"{label:22} {time.perf_counter() - t:.3f}s {len(result)} links")
timed("html.parser, full", lambda: BeautifulSoup(html, "html.parser").find_all("a"))
timed("lxml, full", lambda: BeautifulSoup(html, "lxml").find_all("a"))
only_links = SoupStrainer("a")
timed("lxml + SoupStrainer", lambda: BeautifulSoup(html, "lxml", parse_only=only_links).find_all("a"))
# html.parser, full 1.339s 5000 links (timings vary by machine)
# lxml, full 0.725s 5000 links
# lxml + SoupStrainer 0.582s 5000 links
hrefs = [str(a["href"]) for a in BeautifulSoup(html, "lxml", parse_only=only_links).find_all("a")]
print(hrefs[:2]) # ['/i/0', '/i/1']: plain strings, tree can be freedRemember
- lxml is the fastest parser backend for Beautiful Soup
- SoupStrainer + parse_only builds only the parts you need
- Convert results to plain str so the tree can be garbage-collected
- For extreme throughput, drop down to lxml or selectolax
Watch out for
- Storing NavigableString objects in a long-lived list, keeping every parsed document in memory.