Topolearnfind and find_all are the workhorses of extraction; nearly every scraper is built from them.
find_all(name, attrs, recursive, string, limit, **kwargs) returns a list-like ResultSet of all matching tags below the current one; find() returns the first match or `None`.
Ways to filter:
- By name:
soup.find_all("a"). - By attribute keyword:
soup.find_all(id="main"),soup.find_all(href=True)for "has the attribute". - By class:
soup.find_all("div", class_="price"). The underscore is needed becauseclassis a Python keyword. Matching works on any one of the tag's classes. - Attributes whose names aren't valid Python identifiers, such as
data-id, go in a dict:attrs={"data-id": "42"}. - By text:
string="Next"matches tags whose.stringis exactly that. limit=5stops after five matches;recursive=Falsesearches direct children only.
Calling a tag like a function is shorthand for find_all: soup("a") equals soup.find_all("a").
You can call these methods on any tag to search only its subtree, which is the main way to scope a search: find a container first, then search inside it.
Remember that find() returning None is normal for optional content; soup.find("span").text crashes with AttributeError: 'NoneType' when the span is missing.
Example
from bs4 import BeautifulSoup
html = """<ul id="products">
<li class="item sale" data-id="1"><a href="/p/1">Kettle</a> <span class="price">25</span></li>
<li class="item" data-id="2"><a href="/p/2">Toaster</a> <span class="price">40</span></li>
<li class="item" data-id="3"><a>Blender</a></li>
</ul>"""
soup = BeautifulSoup(html, "html.parser")
items = soup.find_all("li", class_="item")
print(len(items)) # 3
print(soup.find("li", class_="sale").a.string) # Kettle
print([a["href"] for a in soup.find_all("a", href=True)]) # ['/p/1', '/p/2']
print(soup.find(attrs={"data-id": "2"}).a.string) # Toaster
print(soup.find("a", string="Blender")) # <a>Blender</a>
for li in items:
price = li.find("span", class_="price")
print(li.a.string, price.string if price else "n/a")
# Kettle 25 / Toaster 40 / Blender n/aRemember
- find_all returns all matches, find the first or None
- class_= for classes, attrs={...} for names like data-id
- href=True means 'has this attribute'
- Search inside a container tag to scope results
Watch out for
- Chaining .text on find() results without handling None.