HN Debrief

How We Pushed CDC into Postgres

  • Databases
  • Infrastructure
  • Open Source
  • Developer Tools

Snowflake’s post explains how it captures changes from Postgres and mirrors them into Snowflake, pushing part of the change data capture path into Postgres itself instead of relying entirely on an external connector. The point is not that Snowflake invented CDC. It is that warehouse vendors are trying to make OLTP-to-OLAP replication feel native, cheaper to operate, and less like a fragile chain of third-party tools.

If you run Postgres and depend on analytics replicas, expect more vendors to bundle replication into their database products instead of selling it as a separate pipeline. The buying decision is shifting from raw feature lists to operational trust, especially around schema migrations, deletes, updates, and failure recovery.

Discussion mood

Interested and mostly positive about making Postgres-to-warehouse sync feel native, but grounded by scars from unreliable replication products, painful schema-change behavior, and disappointment that the CDC implementation is not fully open.

Key insights

  1. 01

    Columnar apply work is the real bottleneck

    Applying Postgres changes into a columnar system is where this gets ugly. Inserts are easy, but deletes and primary key updates force you to track where rows landed or pay for expensive scans. That reframes Snowflake’s design choice. The Postgres extension can stay lightweight because the heavy reconciliation work happens downstream in Snowflake.

    When you evaluate CDC products, ask where delete and update application actually runs and what metadata it needs to maintain. That answer will tell you more about cost and failure modes than the ingest demo will.

      Attribution:
    • mslot #1
  2. 02

    Zero-ETL products still fail in production

    Aurora Zero-ETL was described as brittle under ordinary operational churn. One account said schema changes could require VACUUM FULL to resume replication, the pipeline sometimes stopped without explanation, and support escalations were slow. Another pointed to a Trino issue alleging random data deletion. The lesson is that vendor-managed replication is not automatically safer than assembled pipelines.

    Treat 'zero ETL' as a claim to verify, not a category to trust. Before committing, test schema migrations, stalled pipelines, and data reconciliation on a nontrivial workload.

      Attribution:
    • bastawhiz #1
    • fock #1
  3. 03

    Users want warehouse sync as a database checkbox

    The strongest product ask was not for another standalone connector. It was for Postgres providers to expose guaranteed mirroring directly to Iceberg or a warehouse the same way they expose pooling or backups. That is a shift in buyer expectation. Replication is turning into baseline database platform functionality, not an integration project teams want to own.

    If you sell databases or data infrastructure, expect customers to push this into the control plane. If you buy, favor vendors that make replication an operational feature with explicit guarantees, not a reference architecture.

      Attribution:
    • hasyimibhar #1 #2
    • jbonatakis #1
    • mslot #1
  4. 04

    Open source pg_lake stops short of CDC

    The article points readers toward pg_lake, but commenters digging for the replication machinery found that the CDC-specific pieces are missing from the public project and referenced through a private repository. That changes how to read the announcement. The interesting part is product capability, not a reusable open source implementation teams can inspect and run themselves.

    Do not assume an adjacent open source repo means the core replication path is available for self-hosting or audit. Check exactly which components are public before you build around the ecosystem story.

      Attribution:
    • jauntywundrkind #1 #2
    • plaur782 #1

Against the grain

  1. 01

    SaaS-only databases are a bad fit for some teams

    For smaller companies with large datasets, managed database pricing can be worse than self-hosting, especially when charges stack on storage and transfer. There is also a control issue. If your team already runs infrastructure comfortably with tools like Ansible and Terraform, a SaaS-only warehouse or database can remove options without solving a pressing problem.

    If data volume is high and your ops burden is already under control, model long-run storage and egress costs before embracing a hosted analytics stack. Keep self-hosted and open deployment options on the table if pricing leverage matters.

      Attribution:
    • arvyy #1
    • cheema33 #1

In plain english

CDC ↩
Change Data Capture, a method for tracking inserts, updates, and deletes in a database and sending those changes to another system.
DMS ↩
AWS Database Migration Service, an Amazon service for moving and replicating database data.
ETL ↩
Extract, Transform, Load, a data pipeline process that moves data from one system to another and often reshapes it along the way.
Iceberg ↩
Apache Iceberg, an open table format for large analytic datasets stored in object storage.
LTAP ↩
A term used for systems that try to combine transactional and analytical processing patterns more tightly, often by sharing storage or reducing copying.
OLAP ↩
Online Analytical Processing, systems optimized for large analytical queries across many rows of data.
OLTP ↩
Online Transaction Processing, databases optimized for frequent small reads and writes that support application workloads.
pg_lake ↩
An open source PostgreSQL extension from Snowflake Labs for integrating Postgres with lakehouse-style storage systems.
Postgres ↩
PostgreSQL, a popular open source relational database.
VACUUM FULL ↩
A PostgreSQL maintenance command that rewrites a table to reclaim space, usually with heavy locking and operational cost.

Reference links

Vendor replication products and services

Snowflake and pg_lake references