Sharing data between processes

Processes don't share memory by default; choosing the right channel decides both correctness and speed.

Processes have separate memory, so you choose how data moves:

Message passing (preferred):

multiprocessing.Queue(): process-safe FIFO; items are pickled. Same API as queue.Queue (put, get), plus close() and join_thread().
Pipe(): a pair of connected endpoints for two processes, faster than a Queue.

Shared memory (for large numeric data):

Value("i", 0) and Array("d", 1000): ctypes objects in shared memory with an optional lock (with v.get_lock():).
multiprocessing.shared_memory.SharedMemory (3.8+): a named raw block that any process can attach to. Wrap it with a NumPy array or memoryview for zero-copy sharing of big buffers. Close it in every process and unlink() it once.

Managers: Manager() starts a server process holding real Python objects (dict, list, Namespace), with proxies in each process. Convenient and flexible, even across machines, but every operation is an IPC round-trip, so it's slow.

Pitfall with queues: a process that has put items on a Queue won't exit until they've been flushed to the pipe. Joining that process before draining the queue can deadlock, so get() results first, then join().

Example

import multiprocessing as mp
from multiprocessing import shared_memory

def producer(q, n):
    for i in range(n):
        q.put(i * i)
    q.put(None)                              # sentinel

def add_one(name, size):
    shm = shared_memory.SharedMemory(name=name)
    for i in range(size):
        shm.buf[i] += 1                      # writes go straight to shared memory
    shm.close()

if __name__ == "__main__":
    q = mp.Queue()
    p = mp.Process(target=producer, args=(q, 5)); p.start()
    print(list(iter(q.get, None)))           # [0, 1, 4, 9, 16]  (drain, then join)
    p.join()

    shm = shared_memory.SharedMemory(create=True, size=4)
    shm.buf[:4] = bytes([1, 2, 3, 4])
    w = mp.Process(target=add_one, args=(shm.name, 4)); w.start(); w.join()
    print(bytes(shm.buf[:4]))                # b'\x02\x03\x04\x05'
    shm.close(); shm.unlink()

Remember

Prefer message passing: multiprocessing.Queue or Pipe
Value, Array and SharedMemory share raw memory without copying
Managers share Python objects via a server process, at IPC cost

Watch out for

Joining a producer process before draining its queue, causing a deadlock.

Check yourself

Which is best for sharing a 2 GB NumPy array read-only across worker processes?

Official docs: docs.python.org/3/library/multiprocessing.shared_memory.html

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks