TopolearnCSS selectors are a compact, familiar language for locating elements, and they map directly from browser dev tools.
soup.select("css selector") returns a list of all matching tags; soup.select_one(...) returns the first or None. They're powered by the soupsieve library, installed with Beautiful Soup.
Selectors you'll use constantly:
div.price,#main,a[href],a[href^="https"](starts with),img[src$=".png"](ends with),[data-id="42"].- Descendant
ul li, childul > li, adjacent siblingh2 + p, general siblingh2 ~ p. - Pseudo-classes:
li:nth-of-type(2),tr:not(.header),p:-soup-contains("Price")for text matching (a soupsieve extension). - Grouping:
h1, h2, h3.
Like find_all, select works on any tag to scope the search: card.select_one(".price").
When to prefer which? select shines for structural queries (child of, nth, attribute prefix) and when you can copy a selector from the browser's developer tools; find_all is more convenient with Python values like regex or functions. Many scrapers mix both.
A caution about selectors copied from dev tools: they're often absurdly specific (div:nth-child(3) > div > span) and break with the next redesign. Prefer selectors based on stable classes, ids or data attributes.
Example
from bs4 import BeautifulSoup
html = """<article id="post">
<h2>Specs</h2><p>Weight: 1.2 kg</p><p>Color: red</p>
<ul class="links"><li><a href="https://x.io/a">A</a></li>
<li><a href="/b">B</a></li><li class="ad"><a href="https://ads.io">Ad</a></li></ul>
</article>"""
soup = BeautifulSoup(html, "html.parser")
print(soup.select_one("#post > h2").string) # Specs
print([p.string for p in soup.select("h2 ~ p")]) # ['Weight: 1.2 kg', 'Color: red']
print([a["href"] for a in soup.select('ul.links li:not(.ad) a[href^="https"]')])
# ['https://x.io/a']
print(soup.select_one('p:-soup-contains("Color")').string) # Color: red
print(soup.select_one(".missing")) # NoneRemember
- select() returns a list, select_one() the first match or None
- Attribute operators ^= $= *= and combinators > + ~ are supported
- :-soup-contains() matches by text
- Prefer stable classes, ids and data attributes over positional selectors
Watch out for
- Pasting long nth-child selectors from dev tools that break on the next layout change.