collections and other stdlib gems

Python · Practical · 10 min

These containers turn ten lines of bookkeeping into one and are fast, tested building blocks.

The collections module provides specialised containers that come up in real code daily.

Counter counts hashable items: Counter(words) gives a dict of item to count, most_common(n) returns the top n, and counters support +, - and update(). Missing keys read as 0 instead of raising.

defaultdict(factory) calls factory() to create a value the first time a missing key is accessed. defaultdict(list) makes grouping trivial: groups[key].append(item). Only d[key] triggers the factory; d.get(key) and key in d don't insert anything.

deque ("deck") is a double-ended queue with O(1) append, appendleft, pop and popleft — use it for queues and BFS instead of list.pop(0). With maxlen=n it keeps only the last n items, perfect for "recent history" buffers.

namedtuple("Point", "x y") creates a tuple subclass with named fields: lightweight, immutable, still unpackable and indexable. The typed equivalent, typing.NamedTuple, uses class syntax with annotations and is usually nicer to read.

Rounding out the module, ChainMap layers several dicts (e.g. CLI args over environment over defaults), and OrderedDict survives mainly for its move_to_end method, since plain dicts are ordered.

Neighbouring modules worth knowing: itertools for iterator plumbing, functools (lru_cache/cache, partial, reduce), heapq for priority queues, bisect for sorted lists, datetime and zoneinfo for time, json, csv, re and logging.

Example

from collections import Counter, defaultdict, deque
from typing import NamedTuple

words = "to be or not to be that is the question".split()
counts = Counter(words)
print(counts.most_common(2))     # [('to', 2), ('be', 2)]
print(counts["missing"])         # 0

by_length = defaultdict(list)
for w in words:
    by_length[len(w)].append(w)
print(by_length[2])              # ['to', 'be', 'or', 'to', 'be', 'is']

recent = deque(["home", "docs", "faq", "blog"], maxlen=3)
print(list(recent))              # ['docs', 'faq', 'blog']
print(recent.popleft())          # docs

class Point(NamedTuple):
    x: float
    y: float = 0.0

p = Point(3, 4)
x, y = p
print(p, p.x, p[1])              # Point(x=3, y=4) 3 4

Remember

  • Counter counts, defaultdict groups, deque queues
  • deque(maxlen=n) keeps a rolling window
  • typing.NamedTuple gives readable immutable records

Watch out for

  • Using list.pop(0) as a queue, which is O(n) per pop.

Check yourself

  1. With `d = defaultdict(int)`, what does `"x" in d` return after evaluating `d["x"]` once?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: docs.python.org/3/library/collections.html