TopolearnHot loops over millions of items benefit from pushing work into C-implemented iterators.
itertools functions, map, filter, zip, sum and friends are implemented in C. When a pipeline uses only C-level callables, the whole loop runs without executing Python bytecode per item, which can be several times faster than a generator expression.
Consequences:
map(int, strings)ormap(operator.mul, xs, ys)is fast: both the loop and the function are in C.map(lambda x: x * 2, xs)loses most of the advantage, since the lambda runs as Python code; a comprehension is then just as fast and more readable.- The
operatormodule (itemgetter,attrgetter,mul) provides C-level callables for common lambdas. sum(map(operator.mul, xs, ys))computes a dot product without a Python-level loop;math.sumprod(3.12+) is faster still.- Consuming with
deque(it, maxlen=0),sum,maxor"".joinkeeps the consumer in C too.
Laziness itself has a cost: each stage adds a little per-item overhead. For small data, a list comprehension that materialises results is often fastest. For large data, laziness wins on memory and often on cache behaviour.
Measure with timeit on realistic sizes, and remember that for numeric arrays, NumPy's vectorised operations beat any iterator pipeline by an order of magnitude.
Note: copying, deep-copying and pickling itertools objects was deprecated in 3.12 and removed in 3.14; don't rely on it.
Example
import math
import operator
import timeit
xs = list(range(1, 200_001))
ys = list(range(200_000, 0, -1))
t_genexp = timeit.timeit(lambda: sum(x * y for x, y in zip(xs, ys)), number=10)
t_map_op = timeit.timeit(lambda: sum(map(operator.mul, xs, ys)), number=10)
t_sumprod = timeit.timeit(lambda: math.sumprod(xs, ys), number=10)
print(t_map_op < t_genexp, t_sumprod < t_map_op) # True True
words = [str(i) for i in range(100_000)]
t_lambda = timeit.timeit(lambda: list(map(lambda s: int(s), words)), number=10)
t_builtin = timeit.timeit(lambda: list(map(int, words)), number=10)
print(t_builtin < t_lambda) # True
print(sum(map(operator.mul, [1, 2, 3], [4, 5, 6])), math.sumprod([1, 2, 3], [4, 5, 6])) # 32 32Remember
- Pipelines made only of C callables avoid per-item Python bytecode
mapwith a lambda is no faster than a comprehension; useoperatorfunctions- Measure; for numeric arrays, NumPy beats iterator pipelines
Watch out for
- Rewriting comprehensions as
map(lambda ...)expecting a speedup.