Output: formatters, prettify and entities

Beautiful Soup · Expert · 10 min

Generating HTML for emails, templates or diffs needs faithful, predictable output rather than a re-indented approximation.

There are several ways to turn a tree back into markup:

  • str(tag) or tag.decode(): compact HTML as parsed (after repairs). Whitespace text nodes are preserved.
  • tag.prettify(): one tag per line, indented. Meant for reading: it changes whitespace, which can change rendering of inline elements and <pre> content, and it makes diffs noisy. Don't use it to write documents back.
  • tag.encode("utf-8"): bytes.

Formatters control escaping of text and attributes, via formatter= on decode, prettify or encode:

  • "minimal" (default): escapes only &, < and >, enough for valid output.
  • "html": converts characters to named HTML entities where possible (é becomes &eacute;).
  • "html5": like "html", but writes void elements HTML5-style (<br> instead of <br/>) and leaves ampersands unescaped where HTML5 allows it.
  • None: no escaping at all, which can produce invalid or unsafe HTML.
  • An HTMLFormatter instance for full control (indentation, void elements). Note that a bare HTMLFormatter() does no entity escaping unless you pass entity_substitution.

Remember that entities are decoded on parsing: &nbsp; becomes \xa0 in the text, and the default formatter writes it back as a raw character, not as &nbsp;. If output must keep entities, use formatter="html".

Attribute order is preserved as parsed, and Beautiful Soup quotes attribute values with double quotes, switching to single quotes when the value contains double quotes.

Example

from bs4 import BeautifulSoup

soup = BeautifulSoup("<p>Caf&eacute; &amp; bar&nbsp;<br>R&amp;D <b>now</b></p>", "html.parser")
p = soup.p

print(repr(p.get_text()))            # 'Café & bar\xa0R&D now'
print(p.decode())                    # minimal: <p>Café &amp; bar <br/>R&amp;D <b>now</b></p>
print(p.decode(formatter="html"))    # <p>Caf&eacute; &amp; bar&nbsp;<br/>R&amp;D <b>now</b></p>
print(p.decode(formatter="html5"))   # <p>Caf&eacute; & bar&nbsp;<br>R&D <b>now</b></p>
print(p.prettify())
# <p>
#  Café &amp; bar
#  <br/>
#  R&amp;D
#  <b>
#   now
#  </b>
# </p>

Remember

  • str()/decode() keep the parsed structure; prettify() is for reading
  • Default 'minimal' formatter escapes only &, < and >
  • formatter='html' writes named entities, 'html5' also uses <br> style
  • Entities are decoded on parse; choose a formatter to write them back

Watch out for

  • Saving prettify() output as the cleaned document and changing whitespace-sensitive content.

Check yourself

  1. After parsing '&nbsp;', how does str(soup) write it with the default formatter?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: www.crummy.com/software/BeautifulSoup/bs4/doc/#output-formatters

Builds on

Unlocks

Nothing builds on this yet.