TopolearnMojibake (garbled accented characters) is the most common silent data-quality bug in scraped text.
Beautiful Soup always works in Unicode internally. Give it a str and it's used as is. Give it bytes and the Unicode, Dammit component works out the encoding, in roughly this order: an encoding you pass, a byte-order mark, the document's own declaration (<meta charset> or the XML declaration), then a statistical guess using chardet, cchardet or charset-normalizer if one is installed, and finally fallbacks like UTF-8 and Windows-1252.
After parsing, soup.original_encoding tells you what it decided.
Controls:
BeautifulSoup(data, "html.parser", from_encoding="iso-8859-1")when you know better.exclude_encodings=["windows-1252"]to rule out a wrong guess.UnicodeDammit(data).unicode_markupconverts bytes without parsing, andUnicodeDammit.detwingle(data)fixes documents mixing UTF-8 and Windows-1252.
With requests, prefer passing resp.content (bytes) over resp.text: requests decodes using the HTTP header and, for text/html without a charset, defaults to ISO-8859-1, which mangles UTF-8 pages whose charset is only in a <meta> tag.
On output, str(soup) and soup.decode() give Unicode; soup.encode("utf-8") or soup.prettify("utf-8") give bytes, and Beautiful Soup updates the document's meta charset to match.
Example
from bs4 import BeautifulSoup, UnicodeDammit
page = '<meta charset="utf-8"><p>Café crème, naïve résumé</p>'.encode("utf-8")
soup = BeautifulSoup(page, "html.parser") # bytes in: detection
print(soup.original_encoding, soup.p.string) # utf-8 Café crème, naïve résumé
# What resp.text does for text/html without a charset header:
wrong = page.decode("iso-8859-1")
print(BeautifulSoup(wrong, "html.parser").p.string) # Café crème, naïve résumé
legacy = "Grüße".encode("cp1252") # no declaration at all
dammit = UnicodeDammit(legacy, ["cp1252"])
print(dammit.unicode_markup, dammit.original_encoding) # Grüße cp1252
print(soup.p.encode("utf-8")) # b'<p>Caf\xc3\xa9 cr\xc3\xa8me, na\xc3\xafve r\xc3\xa9sum\xc3\xa9</p>'Remember
- Pass bytes so Beautiful Soup can detect the encoding
- soup.original_encoding shows the detected encoding
- from_encoding / exclude_encodings override detection
- resp.text can mis-decode HTML; resp.content is safer
Watch out for
- Feeding resp.text to the parser and storing mojibake like 'Café' in the database.