Reading HTML: tags, attributes, the tree

Every value a scraper extracts is found by its place in the HTML, so you need to read HTML before you can point at data in it.

The page in a response is text written in HTML, a format that marks up content with labels saying what each piece is: a heading, a paragraph, a link, a list. Your browser reads that text and draws the page from it.

The labels are tags: words in angle brackets. <p> opens a paragraph and </p>, with a slash, closes it. A pair of tags plus everything between them is an element:

<p>In stock</p>

Elements sit inside other elements. A list <ul> contains one <li> element per entry, and the whole page sits inside <html>, split into a <head> (information about the page, such as its <title>) and a <body> (what you see). Because of this nesting, a page forms a tree, like a family tree: each element has one parent and can have children. Elements with the same parent are siblings.

An opening tag can carry attributes: extra name="value" settings. A link is <a href="/page-2.html">next</a>; its href attribute holds the URL the link goes to. Two attributes matter a lot for finding data:

class puts a label on elements that belong together. Every price on a shop page might have class="price_color". An element can have several classes, separated by spaces.
id gives one element a name that is unique on the page, like id="content".

Some pages also run programs written in JavaScript, a programming language browsers understand, which can add or change content after the page has arrived. Scrapy doesn't run JavaScript: it sees only the HTML text the server sent. A later lesson covers what to do when the data you want isn't in that text.

To see a page's HTML, right-click it in your browser and choose "View page source". The browser's developer tools (right-click, Inspect) show the same elements as a clickable tree.

Example

<html>
  <head><title>All products | Books to Scrape</title></head>
  <body>
    <div id="content">                          <!-- id: unique name -->
      <article class="product_pod">              <!-- one book; parent of h3 and p -->
        <h3><a href="catalogue/a-light-in-the-attic_1000/index.html"
               title="A Light in the Attic">A Light in the ...</a></h3>
        <p class="price_color">£51.77</p>      <!-- class labels every price -->
      </article>
      <li class="next"><a href="catalogue/page-2.html">next</a></li>
    </div>
  </body>
</html>

Remember

HTML marks content with tags; a pair of tags plus its content is an element
Elements nest into a tree of parents, children and siblings
Attributes like href, class and id sit inside the opening tag
Scrapy sees the HTML as sent, not what JavaScript adds later

Watch out for

Assuming everything you see in the browser is in the HTML the server sent.

Check yourself

In <a href="/page-2.html">next</a>, where is the URL of the next page?
What does class="price_color" on many elements tell you?

Official docs: docs.scrapy.org/en/latest/topics/selectors.html

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks