TopolearnRun-length encoding, splitting logs by session and summarising sorted data all use groupby.
groupby(iterable, key=None) yields (key_value, group_iterator) pairs, where each group contains consecutive items with the same key. It does not look ahead or collect: when the key changes, a new group starts.
So to group all items by a key, sort by the same key first:
for dept, people in groupby(sorted(staff, key=by_dept), key=by_dept):
...Without sorting, groupby on a a b a yields three groups: a, b, a. Sometimes that's exactly what you want (run-length encoding, consecutive events in a log).
The groups are lazy views that share the underlying iterator. Advancing groupby to the next key makes the previous group iterator empty. If you need the groups later, materialise each one immediately, e.g. {k: list(g) for k, g in groupby(...)}.
key is any function; operator.itemgetter("dept") and operator.attrgetter("dept") are fast choices.
When you just want a dict of lists and the input isn't sorted, a defaultdict(list) is simpler and O(n) instead of O(n log n). Use groupby for runs, for streaming already-sorted data (such as database results ordered by the key), or to avoid holding all groups in memory.
Example
from itertools import groupby
from operator import itemgetter
print([(k, len(list(g))) for k, g in groupby("aaabccaa")])
# [('a', 3), ('b', 1), ('c', 2), ('a', 2)]
staff = [{"name": "ann", "dept": "ops"}, {"name": "bob", "dept": "dev"},
{"name": "cy", "dept": "ops"}]
by_dept = itemgetter("dept")
groups = {k: [p["name"] for p in g]
for k, g in groupby(sorted(staff, key=by_dept), key=by_dept)}
print(groups) # {'dev': ['bob'], 'ops': ['ann', 'cy']}
saved = [(k, g) for k, g in groupby("aabb")] # groups not consumed in time
print([(k, list(g)) for k, g in saved]) # [('a', []), ('b', [])]Remember
- groupby groups only consecutive equal keys; sort by the key first to group globally
- Group iterators are invalidated when groupby advances; materialise them immediately
- Use defaultdict(list) for unsorted input when you just need a dict of lists
Watch out for
- Calling groupby on unsorted data and getting the same key in several groups.