TopolearnSometimes a single pass must feed two consumers, such as computing pairs or peeking ahead.
tee(iterable, n=2) returns n independent iterators over the same data. Internally it buffers items that one copy has produced but another hasn't consumed yet.
Rules that follow from that design:
- Don't use the original iterator after teeing. Advancing it directly skips items for all copies without them knowing.
- Memory grows with the gap between the fastest and slowest copy. If one copy is fully consumed before the other starts, tee stores everything, and a plain
list()would be simpler and faster. - tee iterators are not thread-safe; don't consume copies from different threads without a lock.
Good uses keep the copies close together: pairwise used to be implemented as a, b = tee(it); next(b, None); zip(a, b). Another is lookahead: tee off a copy, peek at the next item, and keep the rest.
When you need to iterate a dataset twice and it fits in memory, convert it to a list or tuple. When it doesn't fit, re-create the iterator from its source (re-open the file, re-run the query) rather than teeing.
Example
from itertools import islice, tee
def readings():
for x in [10, 12, 9, 15, 15, 11]:
yield x
a, b = tee(readings())
next(b) # shift the second copy by one
changes = [y - x for x, y in zip(a, b)]
print(changes) # [2, -3, 6, 0, -4]
def peek(it):
it, look = tee(it)
return next(look, None), it
first, rest = peek(iter("xyz"))
print(first, list(rest)) # x ['x', 'y', 'z']
src = iter(range(5))
c1, c2 = tee(src)
next(src) # advancing the original...
print(list(c1), list(islice(c2, 2))) # [1, 2, 3, 4] [1, 2] -- 0 lostRemember
teemakes n iterators that share a buffer of unconsumed items- Never advance the original after teeing
- If one copy runs far ahead, just use a list or re-create the iterator
Watch out for
- Teeing an iterator, consuming one copy completely, then the other, and holding the whole dataset in memory anyway.