TopolearnDifferent parsers build different trees from the same broken HTML, which explains 'it works on my machine' scraping bugs.
Valid HTML parses the same everywhere. Broken HTML, which is most of the web, doesn't: each parser repairs it with its own rules.
- html.parser (standard library): decent speed, no dependencies, lenient but not browser-accurate; doesn't add
<html>/<body>wrappers. - lxml: C-based and the fastest; adds
<html><body>around fragments; its repairs can differ from browsers. - html5lib: implements the HTML5 parsing algorithm, so the tree matches what a browser builds (including
<head>, implicit<tbody>in tables); pure Python and slow.
Example: for <a></p>, html.parser gives <a></a>, lxml gives <html><body><a></a></body></html>, and html5lib gives <html><head></head><body><a><p></p></a></body></html>.
This matters when selectors are written against browser dev tools: the browser inserted a <tbody> that html.parser and lxml don't add, so table > tbody > tr matches nothing. Using html5lib, or selectors that skip tbody (table tr), fixes it.
XML needs lxml's XML mode: BeautifulSoup(data, "xml") (also spelled "lxml-xml"). It preserves case-sensitive tag names, handles namespaces and self-closing tags properly, which HTML parsers don't.
Choose by need: lxml for speed, html5lib for fidelity, html.parser for zero dependencies.
Example
from bs4 import BeautifulSoup
broken = "<a></p>"
for parser in ["html.parser", "lxml", "html5lib"]:
try:
print(f"{parser:12}", BeautifulSoup(broken, parser))
except Exception as e: # FeatureNotFound if not installed
print(f"{parser:12} not installed: {type(e).__name__}")
# html.parser <a></a>
# lxml <html><body><a></a></body></html>
# html5lib <html><head></head><body><a><p></p></a></body></html>
table = "<table><tr><td>1</td></tr></table>"
print(BeautifulSoup(table, "html.parser").select("table > tbody > tr")) # []
print(BeautifulSoup(table, "html.parser").select("table tr")) # [<tr><td>1</td></tr>]
xml = '<Feed><Item id="1"/><Item id="2"/></Feed>'
print([i["id"] for i in BeautifulSoup(xml, "xml").find_all("Item")]) # ['1', '2']Remember
- Parsers agree on valid HTML and differ on broken HTML
- lxml is fastest; html5lib matches browsers; html.parser needs nothing
- Browsers add <tbody>; html.parser and lxml don't
- Use the 'xml' feature (lxml) for XML documents
Watch out for
- Writing selectors with tbody copied from dev tools and getting empty results.