TopolearnStreams give Redis durable, replayable, at-least-once messaging for event processing and work queues.
A stream is an append-only log. XADD orders * item pen qty 2 appends an entry whose ID (* = auto) is <milliseconds>-<sequence>, so entries are time-ordered. XRANGE key - + reads a range, XREAD BLOCK 5000 STREAMS key $ waits for new entries, and XLEN counts them. Unlike Pub/Sub, entries stay until trimmed, so readers can replay history.
Consumer groups distribute entries among workers:
XGROUP CREATE orders billing $ MKSTREAMcreates a group starting at new entries (0to process history),- each worker calls
XREADGROUP GROUP billing worker-1 COUNT 10 BLOCK 5000 STREAMS orders >;>means entries never delivered to this group, and each entry goes to only one consumer in the group, - delivered entries sit in the group's pending entries list (PEL) until the worker calls
XACK.
If a worker crashes, its entries stay pending. Other workers find them with XPENDING and take them over with XAUTOCLAIM after an idle timeout, giving at-least-once delivery. Make handlers idempotent, because an entry may be processed twice.
Several groups can read the same stream independently, like separate subscriptions. Cap growth with XADD key MAXLEN ~ 100000 * ... or XTRIM; the ~ allows efficient approximate trimming. Never trim entries that are still pending if you need them.
Example
redis-cli XGROUP CREATE orders billing '$' MKSTREAM # OK
redis-cli XADD orders '*' item pen qty 2 # "1760700000000-0"
redis-cli XADD orders '*' item pad qty 1 # "1760700000001-0"
redis-cli XREADGROUP GROUP billing w1 COUNT 1 STREAMS orders '>'
# 1) 1) "orders"
# 2) 1) 1) "1760700000000-0"
# 2) 1) "item" 2) "pen" 3) "qty" 4) "2"
redis-cli XPENDING orders billing # 1 pending, owned by w1
redis-cli XACK orders billing 1760700000000-0 # (integer) 1
# w1 died holding an entry? another worker claims entries idle > 60 s
redis-cli XAUTOCLAIM orders billing w2 60000 0-0 COUNT 10
redis-cli XADD orders MAXLEN '~' 100000 '*' item ink qty 5 # append and trimRemember
- XADD appends entries with time-based IDs; entries persist until trimmed
- Consumer groups split entries among consumers; > reads new entries
- Entries stay pending until XACK; XAUTOCLAIM recovers from crashed workers
- Delivery is at-least-once: make handlers idempotent
Watch out for
- Never calling XACK: the pending list grows forever and memory leaks.
- Letting a stream grow unbounded without MAXLEN or XTRIM.