🧭 Google Cloud Pub/Sub: Event-Driven Messaging for Modern Applications

In today’s digital world, real-time communication between systems is essential. From IoT sensors streaming data every second to microservices interacting seamlessly — modern systems rely on event-driven architecture.

That’s where Google Cloud Pub/Sub comes in.

Pub/Sub (Publish/Subscribe) is a fully managed messaging system that allows applications to send and receive messages asynchronously. It enables real-time data streaming, event-driven workflows, and loose coupling between components.

In this article, we’ll explore:

• What Pub/Sub is and how it works
• Architecture and workflow with a ****
• 3 real-world examples (Python, streaming, analytics)
• Tips for remembering concepts (for interview/exams)
• Why it’s important for data engineers and developers

Let’s begin!

⚙️ What Is Google Cloud Pub/Sub?

Pub/Sub stands for Publish/Subscribe — a messaging pattern where:

• Publishers send messages to a topic.
• Subscribers receive those messages asynchronously through subscriptions.

It acts like a message bus that connects producers and consumers without requiring them to know each other.

🧩 Pub/Sub = “Producer publishes → Broker (Topic) → Consumer subscribes”

🌐 Key Features

🧩 Pub/Sub Architecture

To understand Pub/Sub, let’s visualize how publishers, topics, and subscribers work together.

🧭 (Architecture Representation)

            ┌─────────────────────────────┐
            │         Publisher            │
            │  (sends messages to topic)   │
            └──────────────┬───────────────┘
                           │
                           ▼
                 ┌──────────────────┐
                 │      Topic        │
                 │ (Message Broker)  │
                 └──────────────────┘
                           │
            ┌──────────────┴───────────────┐
            ▼                              ▼
  ┌──────────────────┐            ┌──────────────────┐
  │  Subscription A  │            │  Subscription B  │
  │ (Pull Subscriber)│            │ (Push Subscriber)│
  └──────────────────┘            └──────────────────┘
            │                              │
            ▼                              ▼
    ┌───────────────┐            ┌─────────────────┐
    │  Consumer App │            │  Cloud Function  │
    └───────────────┘            └─────────────────┘

💡 How Pub/Sub Works

1. Publisher creates a topic and publishes messages.
2. Subscribers attach to the topic through subscriptions.
3. When a message is published, Pub/Sub stores it temporarily.
4. Subscribers pull messages (or receive via push) asynchronously.
5. Once processed, the subscriber acknowledges the message.
6. Unacknowledged messages are retried until confirmed.

This architecture ensures reliability, scalability, and decoupling.

⚙️ 3 Core Components

1️⃣ Topic

A named resource where publishers send messages. Example: projects/myproject/topics/user-events

2️⃣ Subscription

A connection to a topic that delivers messages to subscribers. Example: projects/myproject/subscriptions/user-activity-sub

3️⃣ Message

Data payload sent by the publisher (e.g., JSON, text, binary).

🧮 Example Set 1: Basic Pub/Sub (Python SDK)

Let’s start with Python examples demonstrating how Pub/Sub works.

🧩 Example 1: Create Topic and Publish a Message

from google.cloud import pubsub_v1

project_id = "my-gcp-project"
topic_id = "user-notifications"

publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(project_id, topic_id)

# Create a topic (if not exists)
topic = publisher.create_topic(request={"name": topic_path})
print(f"Topic created: {topic.name}")

# Publish a message
data = "New user registered!"
future = publisher.publish(topic_path, data.encode("utf-8"))
print(f"Message published: {future.result()}")

🧠 Concept: A publisher creates a topic and sends messages to it. Pub/Sub handles the delivery.

🧩 Example 2: Create a Subscriber (Pull Mode)

from google.cloud import pubsub_v1

project_id = "my-gcp-project"
subscription_id = "user-notifications-sub"

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(project_id, subscription_id)

def callback(message):
    print(f"Received: {message.data.decode('utf-8')}")
    message.ack()

subscriber.subscribe(subscription_path, callback=callback)

print("Listening for messages on:", subscription_path)
import time
while True:
    time.sleep(60)

💡 Explanation: The subscriber pulls messages from the subscription and acknowledges them after processing.

🧩 Example 3: Push Subscription (Webhook Receiver)

# Flask app to receive messages
from flask import Flask, request

app = Flask(__name__)

@app.route("/pubsub/push", methods=["POST"])
def pubsub_push():
    envelope = request.get_json()
    message = envelope["message"]
    print(f"Received message: {message['data']}")
    return "OK", 200

if __name__ == "__main__":
    app.run(port=8080)

🔄 Explanation: In push mode, Pub/Sub delivers messages to a URL endpoint (e.g., Cloud Function or API).

🔄 Example Set 2: Event-Driven Architecture

These examples show how Pub/Sub powers real-world event-driven apps.

🧩 Example 1: IoT Sensor Data Streaming

Publisher (IoT Device)

import json, random
from google.cloud import pubsub_v1

publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path("my-project", "iot-sensor-data")

for i in range(5):
    message = json.dumps({"device_id": "sensor_1", "temperature": random.uniform(25, 35)})
    publisher.publish(topic_path, message.encode("utf-8"))
    print("Published:", message)

Subscriber (Data Processor)

def callback(message):
    data = json.loads(message.data.decode("utf-8"))
    print(f"Device: {data['device_id']} Temp: {data['temperature']}")
    message.ack()

🧠 Use Case: IoT sensors continuously stream temperature data to Pub/Sub for processing or alerting.

🧩 Example 2: E-Commerce Order Event

When a customer places an order, a Pub/Sub message triggers multiple downstream services:

• Email notification
• Inventory update
• Analytics tracking

order_event = {
    "order_id": "A12345",
    "user_id": "U5678",
    "status": "Placed",
    "timestamp": "2025-10-22T10:00:00Z"
}

publisher.publish(topic_path, json.dumps(order_event).encode("utf-8"))

Each microservice subscribes to the order-events topic independently.

🧩 Example 3: Real-Time Analytics Pipeline

Pub/Sub → Dataflow → BigQuery

# Dataflow reads messages from Pub/Sub, transforms them, and writes to BigQuery
python -m apache_beam.examples.streaming_wordcount \
  --input_topic=projects/my-project/topics/stream-data \
  --output_table=my_dataset.analytics_results

🧠 Concept: Pub/Sub provides the input stream, while Dataflow performs real-time transformations.

🧮 Example Set 3: Advanced Integrations

🧩 Example 1: Pub/Sub with Cloud Functions

You can trigger a Cloud Function automatically when a new message arrives.

def process_message(event, context):
    import base64
    message = base64.b64decode(event['data']).decode('utf-8')
    print(f"Processed event: {message}")

🧩 Deployment Command:

gcloud functions deploy process_message \
--trigger-topic=my-topic --runtime=python39

Use Case: Automatically process new user registrations, logins, or payments.

🧩 Example 2: Pub/Sub + BigQuery + Dataflow

Architecture: Pub/Sub (Ingest) → Dataflow (Transform) → BigQuery (Store)

• Pub/Sub collects streaming data
• Dataflow enriches or aggregates it
• BigQuery provides analytics

This trio forms a real-time analytics pipeline used in finance, IoT, and e-commerce.

🧩 Example 3: Pub/Sub Dead Letter Topic (Error Handling)

from google.cloud import pubsub_v1

publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path("my-project", "main-topic")
dl_topic_path = publisher.topic_path("my-project", "dead-letter-topic")

# Create a subscription with dead-letter policy
subscription = pubsub_v1.types.Subscription(
    name="projects/my-project/subscriptions/main-sub",
    topic=topic_path,
    dead_letter_policy=pubsub_v1.types.DeadLetterPolicy(
        dead_letter_topic=dl_topic_path,
        max_delivery_attempts=5
    )
)

publisher.create_topic(request={"name": dl_topic_path})
subscriber = pubsub_v1.SubscriberClient()
subscriber.create_subscription(request={"subscription": subscription})

Purpose: If a message fails processing after multiple attempts, it’s redirected to the dead-letter topic for debugging.

🧠 How to Remember Pub/Sub Concepts (Interview & Exam Tips)

🎯 Mnemonic: “TMS” → Topic, Message, Subscriber

📋 Flashcard Q&A

🚀 Why It’s Important to Learn Pub/Sub

🧩 Common Mistakes & Best Practices

🔍 Real-World Use Cases

🧭 Summary

Google Cloud Pub/Sub is a powerful event-driven messaging system that connects producers and consumers asynchronously.

It enables: ✅ Real-time event processing ✅ Decoupled architecture ✅ Seamless integration with Dataflow, BigQuery, and Cloud Functions ✅ Reliable and scalable message delivery

Pub/Sub is the backbone of event-driven systems on Google Cloud — essential for IoT, analytics, and microservice-based applications.

🧠 Final Thoughts

Learning Pub/Sub gives you an edge as a data engineer, cloud architect, or backend developer. It teaches you how modern event-driven systems work — a crucial skill in the era of real-time data.

“If BigQuery is the brain of analytics, then Pub/Sub is the heartbeat of real-time data flow.”