TopolearnGenerators process large or infinite data with constant memory and power much of the standard library.
A for loop works with any iterable — an object whose __iter__ returns an iterator. An iterator produces items one by one from __next__ and raises StopIteration when it is exhausted. The built-ins iter() and next() expose this protocol directly. Lists are iterables; a file object or a generator is its own iterator, which is why it can only be consumed once.
A generator function is any function containing yield. Calling it doesn't run the body; it returns a generator object. Each next() resumes execution until the next yield, which hands a value out and pauses with all local state preserved. When the function returns, iteration ends.
Generators are lazy: they compute values only when asked. That lets you:
- process files or streams far larger than memory, one record at a time
- represent infinite sequences (
itertools.count()), taking only what you need - build pipelines where each stage is a generator consuming the previous one
Generator expressions (f(x) for x in xs) are the one-line form.
The itertools module is the toolbox for iterators: islice (slice any iterator), chain, groupby, accumulate, product, pairwise and batched (3.12) among others. Built-ins like zip, map, filter, enumerate and reversed also return lazy iterators.
The main gotcha is exhaustion: iterating a generator a second time yields nothing, silently. If you need the values twice, materialise them with list() or recreate the generator.
Example
from itertools import islice, count
def read_records(lines):
for line in lines:
line = line.strip()
if line and not line.startswith("#"):
yield line.split(",")
raw = ["# header", "ana,31", "", "ben,27"]
records = read_records(raw) # nothing runs yet
print(next(records)) # ['ana', '31']
print(list(records)) # [['ben', '27']]
print(list(records)) # [] -- exhausted
def squares():
for n in count(1): # infinite
yield n * n
print(list(islice(squares(), 5))) # [1, 4, 9, 16, 25]
it = iter([10, 20])
print(next(it), next(it), next(it, "done")) # 10 20 doneRemember
- Iterables give iterators; iterators yield items via
next()until StopIteration - A function with
yieldreturns a lazy generator object - Generators and iterators can be consumed only once
Watch out for
- Iterating the same generator twice and getting nothing the second time.