TopolearnScraped data is only useful once it's structured, typed and ready for CSV, a database or pandas.
Most extraction tasks reduce to: find the repeated container, then build one record per container.
For tables:
- Read headers from
thead th(or the first row), normalized withget_text(strip=True). - Iterate rows from
tbody tr, skipping rows withouttdcells (sub-headers, spacers). - Zip headers with cell texts into a dict.
- Watch for
colspan/rowspan, which break simple zipping; handle or reject them explicitly.
For card lists (products, articles), select each card and pull fields with select_one relative to the card, using a helper that returns None for missing fields instead of crashing.
Clean and convert at extraction time: strip whitespace, remove currency symbols and thousands separators, parse numbers and dates, and keep raw text when parsing fails, so you can investigate rather than silently drop data.
For simple, well-formed tables, pandas.read_html() does all of this in one call (it uses lxml or Beautiful Soup underneath). Beautiful Soup is the tool when tables are irregular, when you need attributes such as links inside cells, or when data isn't a table at all.
Finally, validate: count records, check required fields, and log anomalies. Silent partial scrapes are the classic failure.
Example
from bs4 import BeautifulSoup
html = """<table id="t"><thead><tr><th>City</th><th>Pop.</th><th>Link</th></tr></thead>
<tbody>
<tr><td> Paris </td><td>2,102,650</td><td><a href="/paris">wiki</a></td></tr>
<tr class="spacer"><th colspan="3">Germany</th></tr>
<tr><td>Berlin</td><td>3,878,100</td><td></td></tr>
</tbody></table>"""
soup = BeautifulSoup(html, "html.parser")
headers = [th.get_text(strip=True) for th in soup.select("#t thead th")]
rows = []
for tr in soup.select("#t tbody tr"):
cells = tr.find_all("td")
if not cells:
continue # skip sub-header rows
rec = dict(zip(headers, (td.get_text(strip=True) for td in cells)))
rec["Pop."] = int(rec["Pop."].replace(",", ""))
link = cells[2].find("a")
rec["Link"] = link["href"] if link else None
rows.append(rec)
print(rows)
# [{'City': 'Paris', 'Pop.': 2102650, 'Link': '/paris'}, {'City': 'Berlin', 'Pop.': 3878100, 'Link': None}]Remember
- One record per repeated container (row or card)
- Zip header names with cell texts; skip rows without td
- Clean and convert types at extraction time
- pandas.read_html for simple tables; Beautiful Soup for irregular ones
Watch out for
- Zipping headers with cells on tables using colspan, misaligning every column after it.