TopolearnKnowing the node types explains why comments or script contents show up (or don't) in your text and how to handle them precisely.
Every node in the tree is an instance of a few classes, all in bs4.element:
- `BeautifulSoup`: the document root, itself a subclass of
Tag. - `Tag`: an element with
name,attrsandcontents. - `NavigableString`: text; a subclass of
strwith tree links (.parent,.next_sibling), so you can navigate from a string back to its tag. - Special string subclasses: `Comment` (
<!-- ... -->), `CData`, `ProcessingInstruction`, `Declaration`, `Doctype`, and for HTML, `Script`, `Stylesheet` and `TemplateString` for the contents of<script>,<style>and<template>.
Because they're all strings, a naive find_all(string=True) returns comments and script code along with visible text. Check types with isinstance(s, Comment), or use get_text(), which by default only includes ordinary NavigableStrings and CData, leaving out comments, scripts and styles. That's why get_text() on a whole page usually gives readable text without JavaScript.
Attribute values are plain strings, or lists for multi-valued attributes like class, rel and headers; multi_valued_attributes=None in the constructor turns that off.
copy.copy(tag) makes a detached deep copy you can modify without touching the original tree, useful for producing cleaned variants.
Example
import copy
from bs4 import BeautifulSoup, Comment
from bs4.element import Script, NavigableString
html = """<div><!-- tracking id 99 --><p>Hello <b>world</b></p>
<script>var x = 1;</script></div>"""
soup = BeautifulSoup(html, "html.parser")
for s in soup.find_all(string=True):
print(type(s).__name__, repr(s))
# Comment ' tracking id 99 '
# NavigableString 'Hello '
# NavigableString 'world'
# NavigableString '\n'
# Script 'var x = 1;'
print(repr(soup.get_text())) # 'Hello world\n': no comment or script
b_text = soup.b.string
print(isinstance(b_text, str), b_text.parent.name) # True b
clean = copy.copy(soup.div)
clean.p.decompose()
print(soup.div.p is not None) # True: original untouchedRemember
- Tag, NavigableString and special string subclasses make up the tree
- NavigableString is a str that knows its place in the tree
- get_text() skips comments, scripts and styles by default
- copy.copy(tag) gives a detached copy for safe edits
Watch out for
- Collecting find_all(string=True) as 'visible text' and including comments and JavaScript.