TopolearnEvery Beautiful Soup task starts by parsing markup into a navigable tree, and picking the parser explicitly avoids surprises.
Beautiful Soup turns HTML or XML, even badly broken HTML, into a tree of Python objects you can search and navigate. Install it as pip install beautifulsoup4 and import it from the bs4 package (the old BeautifulSoup package is version 3, long dead).
Create a soup with BeautifulSoup(markup, "html.parser"). The markup can be a string, bytes or an open file. The second argument chooses the parser:
"html.parser": Python's built-in parser, no extra install."lxml": much faster, needspip install lxml."html5lib": parses exactly like a browser, slowest.
Always pass the parser explicitly. If you omit it, Beautiful Soup picks the best one installed and warns, which means the same code can produce different trees on different machines.
The resulting BeautifulSoup object is the root of the tree. Useful first steps:
soup.titlegives the first<title>tag;soup.title.stringits text.soup.prettify()returns the document re-indented, one tag per line, for reading.print(soup)orstr(soup)gives the (repaired) HTML back.
Beautiful Soup doesn't download pages or run JavaScript; it only parses what you give it. Pair it with an HTTP client like requests for fetching.
Example
from bs4 import BeautifulSoup
html = """
<html><head><title>Daily News</title></head>
<body>
<h1 id="top">Headlines</h1>
<p class="story">Markets rally</p><p class="story">Rain expected</p>
</body></html>
"""
soup = BeautifulSoup(html, "html.parser")
print(soup.title) # <title>Daily News</title>
print(soup.title.string) # Daily News
print(soup.h1["id"]) # top
print(soup.p) # <p class="story">Markets rally</p>
print(soup.prettify()[:40].splitlines()[:3])
# ['<html>', ' <head>', ' <title>']Remember
- pip install beautifulsoup4, import from bs4
- BeautifulSoup(markup, 'html.parser') builds the tree
- Always name the parser explicitly
- It parses only; fetching and JavaScript are not its job
Watch out for
- Omitting the parser argument and getting different results on another machine.