TopolearnChoosing between data structures for millions of items needs real numbers about per-object overhead.
sys.getsizeof(obj) returns the size of an object in bytes, as reported by the object itself plus garbage-collector overhead. It's useful for comparing representations, but it is shallow: for containers it counts the container's own structure (header plus pointer array), not the objects it refers to.
So sys.getsizeof([big_string]) is small regardless of the string's length, and a dict's size doesn't include its keys and values. To estimate a structure's total footprint, walk it recursively, keeping a set of ids to count shared objects once, or use tracemalloc to measure actual allocations.
Useful observations on 64-bit CPython:
- Every object has a header; even
0andTruetake around 24 to 28 bytes. - Lists over-allocate when they grow, so a list built by
appendmay report more than one created at its final size. __slots__classes avoid a per-instance__dict__, which matters for millions of small objects.array.arrayand NumPy arrays store raw values, so a million floats take about 8 MB instead of roughly 32 MB in a list of float objects.
Exact numbers vary between Python versions and builds, so compare relative sizes rather than hard-coding them.
Example
import array
import sys
text = "x" * 10_000
print(sys.getsizeof(text) > 10_000) # True
print(sys.getsizeof([text]) < 100) # True: shallow
nums = list(range(100_000))
deep = sys.getsizeof(nums) + sum(sys.getsizeof(n) for n in nums)
packed = sys.getsizeof(array.array("q", nums))
print(deep > 3 * packed) # True
class P:
def __init__(self, x): self.x = x
class S:
__slots__ = ("x",)
def __init__(self, x): self.x = x
p, s = P(1), S(1)
print(sys.getsizeof(s) < sys.getsizeof(p) + sys.getsizeof(p.__dict__)) # TrueRemember
sys.getsizeofis shallow: containers don't include their elements- Every Python object carries a header of a few dozen bytes
- Use
tracemallocor a recursive walk for total memory
Watch out for
- Concluding that a dict of large strings is small because
sys.getsizeof(d)returned a few hundred bytes.