TopolearnPub/Sub is the event backbone between services, functions and data pipelines on Google Cloud.
Pub/Sub is a global, serverless messaging service. Publishers send messages (bytes plus string attributes) to a topic. Each subscription on the topic receives its own copy of every message, so you fan out by adding subscriptions.
Subscription types:
- Pull: your workers fetch messages, typically with the client library's streaming pull. Best for throughput and control.
- Push: Pub/Sub POSTs each message to an HTTPS endpoint, such as a Cloud Run service, optionally with an OIDC token.
- Export subscriptions write directly to BigQuery or Cloud Storage.
Delivery is at-least-once. A subscriber must acknowledge each message within the ack deadline (default 10 seconds, extended automatically by client libraries); otherwise it is redelivered. Make handlers idempotent. Unacked messages are retained for 7 days by default.
Reliability features:
- Dead-letter topics after N failed delivery attempts (5 to 100), so poison messages stop looping.
- Retry policy with exponential backoff.
- Ordering keys: messages with the same key are delivered in order (enable on the subscription).
- Exactly-once delivery for pull subscriptions in one region, if duplicates are truly unacceptable.
- Filters so a subscription only receives messages whose attributes match.
Pub/Sub is not a log you replay from offset 0 by default; for that, enable retention of acknowledged messages and use seek, or consider Managed Kafka.
Example
# pip install google-cloud-pubsub
from concurrent.futures import TimeoutError
from google.cloud import pubsub_v1
project = "shop-dev-4821"
publisher = pubsub_v1.PublisherClient()
topic = publisher.topic_path(project, "orders")
future = publisher.publish(topic, b'{"id": 42}', origin="web") # attributes as kwargs
print("published", future.result()) # message ID
subscriber = pubsub_v1.SubscriberClient()
sub = subscriber.subscription_path(project, "orders-worker")
def handle(message):
print(message.data, message.attributes["origin"]) # b'{"id": 42}' web
message.ack() # nack()/no ack -> redelivery
streaming = subscriber.subscribe(sub, callback=handle)
with subscriber:
try:
streaming.result(timeout=30)
except TimeoutError:
streaming.cancel()
streaming.result()Remember
- Each subscription gets every message; fan out by adding subscriptions
- At-least-once: ack within the deadline and keep handlers idempotent
- Use dead-letter topics for poison messages
Watch out for
- Acknowledging before processing finishes, so a crash loses the message.