🚀 Streaming PostgreSQL Changes to BigQuery using Cloud Run Jobs + Cloud Scheduler 🔄

Publicado el Publicado en Planet

This lightweight Change Data Capture (CDC) pipeline streams PostgreSQL logical replication events to BigQuery — no Debezium, no Kafka, just native GCP tools.

🧩 Architecture Overview

[Cloud SQL (PostgreSQL)]
   └─ Logical replication via wal2json plugin
        ↓
[Cloud Scheduler] (runs every 12 minutes)
   └─ Triggers →
[Cloud Run Job (custom CDC listener)]
   └─ Pulls logical changes from slot
   └─ Publishes change events (JSON) →
[Cloud Pub/Sub Topic]
   ↓
[Dataflow (Apache Beam Flex Template)]
   ↓
[BigQuery]

đź“Ś GitHub: github.com/HenryXiloj/demos-gcp/tree/main/pg-cdc-to-bq-streaming

đź’ˇ Key Highlights

âś… CDC listener written in Python using psycopg2 with logical replication

âś… Deployed as a Cloud Run Job for short-lived, stateless execution

âś… Orchestrated by Cloud Scheduler (runs every 12 minutes)

âś… Publishes change events as JSON to Pub/Sub

âś… Real-time ingestion into BigQuery using Apache Beam (Dataflow Flex Template)

âś… Includes graceful shutdown with SIGTERM handling

🔄 Why Use Cloud Run Jobs?

  • Precise control over frequency & duration
  • No continuously running services
  • Fully stateless and pay-as-you-go
  • No external brokers or connectors required

⚖️ Compared to Traditional CDC (e.g., Debezium/Kafka)

🔹 No always-on infrastructure

🔹 No Zookeeper or Kafka to maintain

🔹 Native GCP integration for IAM, networking, and monitoring

🔹 Lower cost, easier to secure and deploy

⚡ Performance & Scalability

  • Handles thousands of changes per minute
  • Typical latency: 10–15 minutes
  • Scales with workload by adjusting Cloud Scheduler frequency

This design offers full control, better security posture, and serverless simplicity, making it ideal for event-driven analytics pipelines.

👉 Full implementation:

<

p class=»ember-view reader-text-block__paragraph» id=»ember2618″ style=»background: none 0% 0% / auto repeat scroll padding-box border-box rgb(255, 255, 255); border-color: rgba(0, 0, 0, 0.9); border-image: none 100% / 1 / 0 stretch; border-style: none; border-width: 0px; box-sizing: inherit; color: rgba(0, 0, 0, 0.9); font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", "Fira Sans", Ubuntu, Oxygen, "Oxygen Sans", Cantarell, "Droid Sans", "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Lucida Grande", Helvetica, Arial, sans-serif; font-size: 16px; line-height: 1.5; margin: 0px 0px 32px; padding: 0px; pointer-events: all; vertical-align: baseline;»>đź”— github.com/HenryXiloj/demos-gcp/tree/main/pg-cdc-to-bq-streaming