Topolearn

Learn Beautiful Soup

Parse, search and edit HTML and XML in Python, even when it's badly broken

15 lessons · 3 hours · 4 levels · covers Beautiful Soup 4.15

Beautiful Soup (the bs4 package) parses HTML and XML into a tree of Python objects that you can search with find_all or CSS selectors, navigate and modify. It's the most popular tool for web scraping and for cleaning or transforming HTML, and it copes gracefully with the malformed markup found on real websites.

Sign in free to track progressOfficial docs ↗

Before you start

Beautiful Soup builds on these; the tree links to the exact lessons you need.

PythonEssentials level

How deep do you want to go?

1. EssentialsRead and write simple code, do the everyday basics4 lessons · 38 min in total
2. PracticalUse it productively at work, with the usual idioms8 lessons · 1 hour in total
3. AdvancedThe less common features, performance and trade-offs12 lessons · 2 hours in total
4. ExpertInternals and the edge cases experts know15 lessons · 3 hours in total

The Beautiful Soup tree

Beautiful SoupParsing HTML into a soupParsing HTML into asoupTags, attributes and textTags, attributes andtextSearching with find and find_allSearching with findand find_allCSS selectors with select()CSS selectors withselect()Navigating the treeNavigating the treeAdvanced filters: regex, lists and functionsAdvanced filters:regex, lists and fun…Scraping real pages with requestsScraping real pageswith requestsExtracting tables and lists into dataExtracting tables andlists into dataStrings and f-stringsStrings and f-stringsModules and importsModules and importsDictionariesDictionariesDefining functionsDefining functionsfor and while loopsfor and while loops
Essentials and Practical shown (8 of 15 lessons); the full list is below. Click a lesson to read it.

All 15 lessons, in order

Parsing basics

  1. Parsing HTML into a soup · Essentials · 8 min

    Install bs4, turn an HTML string into a BeautifulSoup tree with an explicit parser, and look at it.

  2. Tags, attributes and text · Essentials · 10 min

    Read a tag's name, attributes and text with ['attr'], .get(), .attrs, .string and get_text().

Searching and navigating

  1. Searching with find and find_all · Essentials · 10 min

    Search the tree with find() and find_all() by tag name, attributes, class_, string and limit.

  2. CSS selectors with select() · Essentials · 10 min

    Use select() and select_one() with CSS selectors for concise, precise queries.

  3. Navigating the tree · Practical · 12 min

    Move around the tree with parent, children, descendants, siblings and the find_parent/find_next_sibling family.

  4. Advanced filters: regex, lists and functions · Practical · 10 min

    Pass regular expressions, lists, True and custom functions to find_all for flexible matching.

Scraping in practice

  1. Scraping real pages with requests · Practical · 12 min

    Fetch pages with requests, parse them, resolve relative links with urljoin, and scrape politely.

  2. Extracting tables and lists into data · Practical · 12 min

    Turn HTML tables and repeated blocks into lists of dicts, cleaning text and converting numbers.

  3. Robust scrapers and dynamic pages · Advanced · 12 min

    Write scrapers that survive layout changes and missing fields, detect JavaScript-rendered content, and know when to switch tools.

Editing, parsers and encodings

  1. Modifying and cleaning HTML · Advanced · 12 min

    Change the tree: remove elements with decompose(), edit attributes and text, create tags, and unwrap or replace elements.

  2. Parsers compared: html.parser, lxml, html5lib · Advanced · 12 min

    How the three HTML parsers differ on broken markup, speed and output, and how to parse XML.

  3. Encodings and Unicode, Dammit · Advanced · 10 min

    How Beautiful Soup detects document encodings from bytes, how to override it, and how output encoding works.

Internals and performance

  1. Performance: SoupStrainer, lxml and memory · Expert · 12 min

    Speed up parsing with lxml and SoupStrainer, reduce memory by releasing trees, and know when Beautiful Soup is the bottleneck.

  2. Object model: Tag, NavigableString and more · Expert · 10 min

    Understand the node classes Beautiful Soup creates (Tag, NavigableString, Comment, Script, Doctype) and how they affect text extraction.

  3. Output: formatters, prettify and entities · Expert · 10 min

    Control how trees are serialized: str versus prettify, entity formatters, void elements and indentation.