What a web page is made of: HTML

Beautiful Soup works on HTML, so you need to be able to read it before you can pull data out of it.

Every web page you visit is, underneath, a text file. Your browser (Chrome, Firefox, Safari) downloads that text and draws the page from it. The text is written in HTML, a format that marks up content with labels saying what each piece is: a heading, a paragraph, a link, a list.

The labels are tags: words in angle brackets. <p> opens a paragraph and </p>, with a slash, closes it. A pair of tags plus everything between them is an element:

<p>Rain expected tomorrow</p>

Elements sit inside other elements. A list <ul> contains list items <li>, and the whole page sits inside <html>, split into a <head> (information about the page, such as its <title>) and a <body> (what you see). Because of this nesting, a page forms a tree, like a family tree: each element has one parent and can have children. Elements with the same parent are siblings.

An opening tag can carry attributes: extra name="value" settings. A link is <a href="/news">News</a>; its href attribute holds the address the link goes to. Three attributes matter a lot for finding data:

class puts a label on elements so they can be styled alike. Every price on a shop page might have class="price". An element can have several classes, separated by spaces.
id gives one element a name that is unique on the page, like id="main".
data- attributes, like data-id="42", hold extra information a site stores for its own use.

Many pages also include programs written in JavaScript, a programming language browsers run. They can change the page after it has arrived; keep that in mind, because Beautiful Soup only sees the HTML text itself.

To see the HTML of any page, right-click it and choose "View page source".

Example

<html>
  <head>
    <title>Daily News</title>          <!-- shown in the browser tab -->
  </head>
  <body>
    <h1 id="top">Headlines</h1>       <!-- id: unique name -->
    <ul>                               <!-- the list is the parent... -->
      <li class="story">Markets rally</li>   <!-- ...of these two siblings -->
      <li class="story">Rain expected</li>
    </ul>
    <a href="/archive" data-id="7">Older stories</a>
  </body>
</html>

Remember

A web page is HTML text: tags like <p> and </p> mark elements
Elements nest inside each other, forming a tree of parents and children
Attributes such as href, class, id and data-* sit in the opening tag
class labels groups of elements; id names one element

Watch out for

Assuming everything you see in the browser is in the page's HTML; JavaScript may have added it later.

Check yourself

In <li class="story">Rain expected</li>, what is class="story"?
A <ul> contains two <li> elements. How are the two <li> elements related?

Official docs: www.crummy.com/software/BeautifulSoup/bs4/doc/#quick-start

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks