TopolearnNetwork round trips, not Redis itself, are usually the bottleneck; pipelining often gives a 10x speedup.
Each command normally costs a full network round trip: send, wait, receive. At 0.5 ms per round trip, 10,000 sequential SETs take 5 seconds even though Redis executes each in microseconds.
Pipelining sends many commands without waiting for each reply, then reads all replies at once. Every client library supports it: in redis-py r.pipeline(transaction=False), in node-redis by firing commands concurrently or with multi()/execAsPipeline. Throughput jumps because the server processes a stream of commands and the network carries big packets instead of tiny ones.
A pipeline is not atomic on its own: other clients' commands may interleave between yours. redis-py's pipeline() wraps commands in MULTI/EXEC by default; pass transaction=False when you only want batching. Keep batches to a few thousand commands so replies don't pile up in memory on both sides.
Multi-key commands (MGET, MSET, HSET with many fields, variadic SADD) are another way to reduce round trips.
Client hygiene:
- reuse a connection pool (one client object per process),
- set connect and socket timeouts,
- use
decode_responses=Truein Python if you wantstrinstead ofbytes, - for bulk imports use
redis-cli --pipewith a file of commands.
Example
import time
import redis
r = redis.Redis() # one pooled client per process
t = time.perf_counter()
for i in range(10_000):
r.set(f"k:{i}", i) # 10,000 round trips
print("one by one", round(time.perf_counter() - t, 2)) # e.g. 0.7 (local; far more over a network)
t = time.perf_counter()
with r.pipeline(transaction=False) as p:
for i in range(10_000):
p.set(f"k:{i}", i)
p.execute() # few round trips
print("pipelined", round(time.perf_counter() - t, 2)) # e.g. 0.1
print(r.mget("k:1", "k:2")) # [b'1', b'2']Remember
- Round trips dominate latency; pipelines batch them
- Pipelines are not atomic unless wrapped in MULTI/EXEC
- Prefer variadic commands like MGET, MSET, SADD with many members
- Reuse one pooled client and set timeouts
Watch out for
- Opening a new connection per request, paying TCP (and TLS) setup every time.