TopolearnStreaming pipelines and coroutine-style code rest on these generator features.
yield from iterable delegates to another iterable, yielding all its items. With a sub-generator it does more than a loop would: values passed in with send() and exceptions passed with throw() are forwarded to the sub-generator, and the expression result = yield from sub() evaluates to the sub-generator's return value (a generator's return x raises StopIteration(x) under the hood).
Generators are two-way. yield is an expression: received = yield value. gen.send(x) resumes the generator and makes the paused yield evaluate to x. You must start a fresh generator with next(gen) (or send(None)) to advance it to the first yield. This is how coroutines worked before async/await, and those keywords are still implemented on the same machinery.
gen.close() raises GeneratorExit at the paused yield, so finally blocks and with statements inside the generator run their cleanup. A generator that holds a resource (an open file) releases it when closed or garbage collected; close it explicitly if you stop early.
The real everyday win is pipelines: small generator stages composed like Unix pipes, each lazily pulling from the previous. Memory use stays flat regardless of input size. itertools supplies the fittings:
chain.from_iterable,islice,takewhile,dropwhilegroupby(input must be sorted by the key),pairwise,batched(it, n)(3.12, and since 3.13strict=True)teeto split an iterator,accumulate,product,combinations
Example
from itertools import groupby
def read_lines(chunks):
for chunk in chunks:
yield from chunk.splitlines() # delegate to another iterable
def parse(lines):
for line in lines:
user, amount = line.split(",")
yield user, int(amount)
chunks = ["ana,5\nben,3", "ana,7\ncy,1"]
records = sorted(parse(read_lines(chunks))) # lazy until sorted() pulls
for user, rows in groupby(records, key=lambda r: r[0]):
print(user, sum(amount for _, amount in rows)) # ana 12 / ben 3 / cy 1
def averager():
total, count, avg = 0, 0, None
while True:
value = yield avg # receives what send() passes
total, count = total + value, count + 1
avg = total / count
avg = averager()
next(avg) # prime: run to the first yield
print(avg.send(10), avg.send(20)) # 10.0 15.0Remember
yield fromdelegates and returns the sub-generator's return valuesend()makesyieldevaluate to a value; prime withnext()first- Compose lazy pipelines from small generators and itertools
Watch out for
- Using
itertools.groupbyon unsorted data and getting split groups.