TopolearnData often lives next to a label or inside an ancestor, and navigation methods let you reach it relative to something you can find.
Once you have a tag, you can move from it:
- Down:
.contents(list of direct children),.children(iterator),.descendants(every node below, recursively, including strings). - Up:
.parent,.parents(iterator up to the soup). - Sideways:
.next_sibling,.previous_sibling,.next_siblings. - Document order:
.next_element/.previous_elementwalk in parse order, crossing tag boundaries.
The classic gotcha: whitespace between tags is a text node. In <td>a</td>\n<td>b</td>, the first td's .next_sibling is the string "\n", not the next td.
That's why the search-while-navigating methods are usually better:
find_next_sibling("td"),find_next_siblings(...),find_previous_sibling(...).find_parent("table"),find_parents(...).find_next("a"),find_all_next(...): anything later in the document.
They accept the same filters as find, so label.find_next_sibling("span", class_="value") skips whitespace and unrelated tags.
A common scraping pattern: locate a stable label by its text (soup.find("th", string="Price")) and navigate to the value next to it, rather than relying on positions that change.
Example
from bs4 import BeautifulSoup
html = """<table id="specs">
<tr><th>Brand</th><td>Acme</td></tr>
<tr><th>Price</th><td>49 EUR</td></tr>
</table>"""
soup = BeautifulSoup(html, "html.parser")
th = soup.find("th", string="Price")
print(th.find_next_sibling("td").string) # 49 EUR
print(th.parent.name, th.find_parent("table")["id"]) # tr specs
first_row = soup.find("tr")
print(repr(first_row.next_sibling)) # '\n': whitespace text node
print(first_row.find_next_sibling("tr").th.string) # Price
print([c.name for c in first_row.children]) # ['th', 'td']
print([type(n).__name__ for n in soup.table.descendants][:4])
# ['NavigableString', 'Tag', 'Tag', 'NavigableString']Remember
- .children/.descendants go down, .parent/.parents up, siblings sideways
- Whitespace between tags is a text node in .next_sibling
- find_next_sibling / find_parent filter while navigating
- Find a stable label, then navigate to its value
Watch out for
- Using .next_sibling and getting '\n' instead of the next tag.