TopolearnProcessing several files, lists or pages as one stream without copying is a frequent need.
chain(a, b, c) yields every item of a, then of b, then of c. Unlike a + b + c, it doesn't build a new list, works with any iterables (sets, generators, files, ranges), and can mix types.
chain.from_iterable(iterables) takes a single iterable of iterables. It's the standard way to flatten one level of nesting:
list(chain.from_iterable([[1, 2], [3], []])) # [1, 2, 3]Because from_iterable reads the outer iterable lazily, it also works when the sub-iterables are produced on demand, such as pages fetched from an API or lines from many files:
lines = chain.from_iterable(open(p) for p in paths)Differences worth knowing:
chain(*lists)unpacks the outer list eagerly;chain.from_iterable(lists)doesn't, so prefer it for generators or huge outer collections.- Strings are iterables of characters, so
chain("ab", "cd")yields characters. - It flattens exactly one level; deeper nesting needs recursion.
sum(list_of_lists, []) also flattens but is quadratic, since each + copies the accumulated list. Use chain.from_iterable or a comprehension with two for clauses instead.
Example
from itertools import chain
a, b = [1, 2], (3, 4)
print(list(chain(a, b, range(5, 7)))) # [1, 2, 3, 4, 5, 6]
nested = [[1, 2], [3], [], [4, 5]]
print(list(chain.from_iterable(nested))) # [1, 2, 3, 4, 5]
print([x for sub in nested for x in sub]) # [1, 2, 3, 4, 5]
def pages(): # e.g. a paginated API
for page in range(3):
yield [f"item{page}{i}" for i in range(2)]
items = chain.from_iterable(pages())
print(next(items), next(items), next(items)) # item00 item01 item10
print(list(chain("ab", "cd"))) # ['a', 'b', 'c', 'd']Remember
chain(a, b)concatenates iterables lazily without copyingchain.from_iterableflattens one level, lazily, even over generators- Avoid
sum(lists, []); it's quadratic
Watch out for
- Expecting chain.from_iterable to flatten deeply nested lists; it removes only one level.