TopolearnSome workloads need fan-out to many readers, replay and huge backlogs, which ordinary queues handle poorly.
A stream (x-queue-type: stream) is an append-only, replicated log, similar in spirit to a Kafka topic partition. Unlike a queue, consuming does not remove messages. Each consumer chooses where to start reading with x-stream-offset: first, last, next, a numeric offset or a timestamp, and can re-read the history at will.
That makes streams the right tool for:
- Large fan-out: many consumers reading the same messages, without a queue per consumer.
- Replay: rebuilding a cache or a read model from history.
- Huge backlogs: millions of messages stored efficiently on disk.
Data is removed only by retention: x-max-length-bytes and x-max-age (such as 7D), applied by deleting whole segment files.
Streams can be used from AMQP 0-9-1 clients like pika, with some rules: the consumer must set a prefetch (basic_qos) and use manual acks (acks just grant credit; nothing is deleted). For the best throughput use the native stream protocol (port 5552) with the dedicated client libraries, which also support server-side offset tracking and super streams, partitioned streams for scaling out with ordering per partition.
Streams don't support dead-lettering, TTL per message or priorities. Use quorum queues for task processing and streams for event logs.
Example
import pika
ch = pika.BlockingConnection(pika.ConnectionParameters("localhost")).channel()
ch.queue_declare("events", durable=True, arguments={
"x-queue-type": "stream",
"x-max-age": "7D", # retention by age
"x-max-length-bytes": 20_000_000_000, # and by size
})
for i in range(3):
ch.basic_publish("", "events", f"event {i}".encode())
ch.basic_qos(prefetch_count=100) # required for stream consumers
seen = []
def on_msg(c, method, props, body):
seen.append(body.decode()); c.basic_ack(method.delivery_tag)
if len(seen) == 3: c.stop_consuming()
ch.basic_consume("events", on_msg, arguments={"x-stream-offset": "first"})
ch.start_consuming()
print(seen) # ['event 0', 'event 1', 'event 2'] -- still in the streamRemember
- Streams are append-only logs; reading doesn't remove messages
- Consumers pick a start with x-stream-offset and can replay
- Retention by age or size; native protocol and super streams for scale
Watch out for
- Using a stream as a task queue and expecting acked messages to disappear.