TopolearnSharing containers between threads without understanding atomicity leads to rare, data-corrupting races.
Thread safety in CPython has two layers.
Individual operations. deque.append, appendleft, pop and popleft are documented as thread-safe, which is why a deque can serve as a simple lock-free queue between threads. Single dict operations such as d[k] = v, d.get(k) and d.pop(k, None) are effectively atomic as well. In the default build the GIL provides this; the free-threaded 3.13 build (python3.13t) keeps these guarantees by using per-object locks (critical sections) internally.
Compound operations are never atomic. counter[key] += 1 is a read, an add and a write; two threads can read the same old value and one increment is lost. The same applies to if key not in d: d[key] = ..., defaultdict factories that do work, and iterating a dict while another thread modifies it (RuntimeError: dictionary changed size during iteration).
Approaches:
- Protect compound updates with a
threading.Lock. - Use
queue.Queuefor producer/consumer handoff; it adds blocking andtask_done. - Give each thread its own Counter and merge at the end with
+=orupdate, which avoids locking entirely. d.setdefault(k, v)is a single operation, useful for "insert if absent".
Free-threading makes races that were rare under the GIL much more frequent, so code that relied on luck surfaces quickly.
Example
import threading
from collections import Counter, deque
def worker(counts, lock, n):
for _ in range(n):
with lock:
counts["hits"] += 1 # compound op: needs the lock
counts, lock = Counter(), threading.Lock()
threads = [threading.Thread(target=worker, args=(counts, lock, 20_000)) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(counts["hits"]) # 80000
def local_count(chunk, results): # lock-free: count locally, merge later
results.append(Counter(chunk))
results = deque() # deque.append is thread-safe
data = ["x", "y", "x"] * 10_000
chunks = [data[i::4] for i in range(4)]
ts = [threading.Thread(target=local_count, args=(c, results)) for c in chunks]
for t in ts: t.start()
for t in ts: t.join()
print(sum(results, Counter())) # Counter({'x': 20000, 'y': 10000})Remember
- deque append/pop operations are thread-safe; single dict operations are atomic
counter[k] += 1is read-modify-write and can lose updates without a lock- Count per thread and merge, or use locks and
queue.Queue
Watch out for
- Assuming the GIL makes
+=on shared counters safe; increments can be lost, especially on free-threaded builds.