HN Debrief

How We Pushed CDC into Postgres

  • Databases
  • Infrastructure
  • Open Source
  • Developer Tools

Snowflake’s post explains how it captures changes from Postgres and mirrors them into Snowflake, pushing part of the change data capture path into Postgres itself instead of relying entirely on an external connector. The point is not that Snowflake invented CDC. It is that warehouse vendors are trying to make OLTP-to-OLAP replication feel native, cheaper to operate, and less like a fragile chain of third-party tools.

If you run Postgres and depend on analytics replicas, expect more vendors to bundle replication into their database products instead of selling it as a separate pipeline. The buying decision is shifting from raw feature lists to operational trust, especially around schema migrations, deletes, updates, and failure recovery.

Discussion mood

Interested and mostly positive about making Postgres-to-warehouse sync feel native, but grounded by scars from unreliable replication products, painful schema-change behavior, and disappointment that the CDC implementation is not fully open.

Key insights

  1. 01

    Columnar apply work is the real bottleneck

    Applying Postgres changes into a columnar system is where this gets ugly. Inserts are easy, but deletes and primary key updates force you to track where rows landed or pay for expensive scans. That reframes Snowflake’s design choice. The Postgres extension can stay lightweight because the heavy reconciliation work happens downstream in Snowflake.

    When you evaluate CDC products, ask where delete and update application actually runs and what metadata it needs to maintain. That answer will tell you more about cost and failure modes than the ingest demo will.

      Attribution:
    • mslot #1
  2. 02

    Zero-ETL products still fail in production

    Aurora Zero-ETL was described as brittle under ordinary operational churn. One account said schema changes could require VACUUM FULL to resume replication, the pipeline sometimes stopped without explanation, and support escalations were slow. Another pointed to a Trino issue alleging random data deletion. The lesson is that vendor-managed replication is not automatically safer than assembled pipelines.

    Treat 'zero ETL' as a claim to verify, not a category to trust. Before committing, test schema migrations, stalled pipelines, and data reconciliation on a nontrivial workload.

      Attribution:
    • bastawhiz #1
    • fock #1
  3. 03

    Users want warehouse sync as a database checkbox

    The strongest product ask was not for another standalone connector. It was for Postgres providers to expose guaranteed mirroring directly to Iceberg or a warehouse the same way they expose pooling or backups. That is a shift in buyer expectation. Replication is turning into baseline database platform functionality, not an integration project teams want to own.

    If you sell databases or data infrastructure, expect customers to push this into the control plane. If you buy, favor vendors that make replication an operational feature with explicit guarantees, not a reference architecture.

      Attribution:
    • hasyimibhar #1 #2
    • jbonatakis #1
    • mslot #1
  4. 04

    Open source pg_lake stops short of CDC

    The article points readers toward pg_lake, but commenters digging for the replication machinery found that the CDC-specific pieces are missing from the public project and referenced through a private repository. That changes how to read the announcement. The interesting part is product capability, not a reusable open source implementation teams can inspect and run themselves.

    Do not assume an adjacent open source repo means the core replication path is available for self-hosting or audit. Check exactly which components are public before you build around the ecosystem story.

      Attribution:
    • jauntywundrkind #1 #2
    • plaur782 #1

Against the grain

  1. 01

    SaaS-only databases are a bad fit for some teams

    For smaller companies with large datasets, managed database pricing can be worse than self-hosting, especially when charges stack on storage and transfer. There is also a control issue. If your team already runs infrastructure comfortably with tools like Ansible and Terraform, a SaaS-only warehouse or database can remove options without solving a pressing problem.

    If data volume is high and your ops burden is already under control, model long-run storage and egress costs before embracing a hosted analytics stack. Keep self-hosted and open deployment options on the table if pricing leverage matters.

      Attribution:
    • arvyy #1
    • cheema33 #1

In plain english

CDC ↩
Centers for Disease Control and Prevention, the main US public health agency for tracking disease and outbreaks.
DMS ↩
AWS Database Migration Service, an Amazon service for moving and replicating database data.
ETL ↩
Extract, transform, load, a data pipeline process for collecting data, cleaning or reshaping it, and storing it for use.
Iceberg ↩
Apache Iceberg, an open table format for large analytical datasets stored in files such as Parquet on object storage.
LTAP ↩
A term used for systems that try to combine transactional and analytical processing patterns more tightly, often by sharing storage or reducing copying.
OLAP ↩
Online Analytical Processing, database workloads focused on large-scale querying, aggregation, and analysis rather than frequent small transactions.
OLTP ↩
Online Transaction Processing, database workloads focused on many small, fast, reliable read and write transactions such as app backends or shopping carts.
pg_lake ↩
A PostgreSQL-related project mentioned as a way to connect Postgres workloads with lakehouse-style storage.
Postgres ↩
PostgreSQL, a widely used open source relational database.
VACUUM FULL ↩
A PostgreSQL maintenance command that rewrites a table to reclaim space, usually with heavy locking and operational cost.

Reference links

Vendor replication products and services

Snowflake and pg_lake references