TopolearnReal pages need fuzzy matching (any heading, links to PDFs, tags with some attribute) that plain strings can't express.
Wherever find_all accepts a value (the tag name, an attribute, string), you can pass richer filters:
- A compiled regex:
soup.find_all(re.compile(r"^h[1-6]$"))matches all headings;href=re.compile(r"\.pdf$")matches PDF links. The regex is applied withsearch, so anchor it if needed. - A list:
soup.find_all(["h1", "h2"])matches any of them. - `True`: matches anything present:
find_all(True)is every tag;attrs={"data-id": True}is "has data-id". - A function: called with each tag, returning
Trueto keep it. For examplelambda t: t.name == "a" and "download" in t.get("class", []). When passed as an attribute filter (href=func), it receives the attribute value (orNone) instead.
The string= argument also accepts these, matching text nodes: soup.find_all(string=re.compile("total", re.I)) returns the matching strings, from which .parent leads back to the tag.
Functions are the escape hatch for any condition, like "a <td> with exactly two children" or "an element whose text parses as a number". Keep them small and named for readability.
For the same patterns in CSS you'd use attribute operators and :-soup-contains; choose whichever reads clearer.
Example
import re
from bs4 import BeautifulSoup
html = """<h1>Report</h1><h3>Q1</h3><p>Total: 120</p>
<a href="/files/q1.pdf">Q1 PDF</a> <a href="/about">About</a>
<span data-id="7">x</span><span>y</span>"""
soup = BeautifulSoup(html, "html.parser")
print([h.name for h in soup.find_all(re.compile(r"^h[1-6]$"))]) # ['h1', 'h3']
print([a.string for a in soup.find_all("a", href=re.compile(r"\.pdf$"))]) # ['Q1 PDF']
print(soup.find_all("span", attrs={"data-id": True})) # [<span data-id="7">x</span>]
def is_numeric_total(tag):
return tag.name == "p" and re.search(r"\d+$", tag.get_text()) is not None
print(soup.find(is_numeric_total).get_text()) # Total: 120
s = soup.find(string=re.compile("total", re.I))
print(repr(s), s.parent.name) # 'Total: 120' pRemember
- Filters can be strings, regexes, lists, True or functions
- A function filter receives each tag and returns True to keep it
- string= with a regex returns matching text nodes; use .parent
- Regexes are searched, so anchor with ^ and $ when needed
Watch out for
- Passing a regex string instead of re.compile(...), which is matched literally.