TopolearnExtracting data means reading attributes like href and the text inside tags correctly.
Each element in the tree is a Tag. Accessing soup.a returns the first <a> anywhere below; chaining like soup.body.p goes one level of "first match" at a time.
Attributes work like a dict:
tag["href"]returns the value, raisingKeyErrorif absent.tag.get("href")returnsNone(or a default) instead: use it for optional attributes.tag.attrsis the full dict;tag.has_attr("data-id")tests presence.- Multi-valued attributes, most importantly
class, come back as a list:["btn", "primary"].
Text has several accessors:
tag.string: the single string child, orNonewhen the tag contains more than one child (such as text plus a<b>).tag.get_text(): all text inside, concatenated. Useget_text(" ", strip=True)to join pieces with a space and strip whitespace, which is what you usually want.tag.stripped_strings: a generator over non-blank text pieces.
tag.name is the tag name ("a"), and you can rename it by assignment. Entities like & are decoded for you: text is plain Unicode.
Most bugs at this stage are about None: a missing tag (soup.find(...) returning None) or .string being None. Check before chaining.
Example
from bs4 import BeautifulSoup
html = """<div class="card featured" data-id="42">
<a href="/item/42" title="Lamp">Desk <b>lamp</b> & bulb</a>
<span class="price"> 19.99 </span>
</div>"""
soup = BeautifulSoup(html, "html.parser")
card, link = soup.div, soup.a
print(card["class"]) # ['card', 'featured']
print(card["data-id"], card.get("missing")) # 42 None
print(link.attrs) # {'href': '/item/42', 'title': 'Lamp'}
print(link.string) # None: several children
print(link.get_text()) # Desk lamp & bulb
print(soup.find("span").get_text(strip=True)) # 19.99
print(list(card.stripped_strings)) # ['Desk', 'lamp', '& bulb', '19.99']Remember
- tag['x'] raises on missing attributes; tag.get('x') returns None
- class is a list of classes
- .string is None when a tag has several children; use get_text()
- get_text(' ', strip=True) for clean text
Watch out for
- Calling .string on a tag with nested markup and getting None.