HN Debrief

Making Postgres queues scale

  • Infrastructure
  • Databases
  • Developer Tools

The DBOS post is a case for keeping queues inside PostgreSQL instead of reaching for Redis, RabbitMQ, or Kafka by reflex. It walks through the usual recipe for a database-backed job queue, especially `FOR UPDATE SKIP LOCKED`, then claims you can push that pattern to much higher throughput with schema changes, partial indexes, and careful tuning. The big idea is not that Postgres magically became a queue, but that for many application job systems, the simplicity of one transactional store can beat a more distributed setup.

If your queue is tightly coupled to app data and lives inside one operational boundary, Postgres is still a strong default. Before adopting it, pressure-test deletion patterns, vacuum behavior, and write amplification, because those are the failure modes that show up long before headline throughput numbers stop looking good.

Discussion mood

Mostly positive on using Postgres for queues, with a practical tone rather than hype. People largely rejected the old blanket claim that it does not scale, but they were quick to point out that success depends on workload shape, especially continuous row churn, vacuum pressure, and whether the queue is an internal job system or a true cross-service messaging layer.

Key insights

  1. 01

    Dead tuples are the real bottleneck

    Dead tuples from updating and deleting queue rows are what make a Postgres queue go sideways in practice. `SKIP LOCKED` can keep workers from blocking each other, but it does nothing about MVCC churn, stale index entries, and planner blind spots, so performance can collapse quickly under continuous processing even when the locking pattern itself looks sound.

    Load-test with sustained dequeue and completion traffic, not just claim throughput. Watch table bloat, index growth, autovacuum lag, and query plans before you commit to this design.

      Attribution:
    • atombender #1 #2
  2. 02

    Job queue and message bus are different problems

    A Postgres queue fits best when background work belongs to the same application and database. Once the reason for queuing is to decouple services or create a separate architectural boundary, using the primary relational database as the transport starts solving the wrong problem, even if it can handle the raw job volume.

    Be explicit about why you want a queue. If the need is asynchronous work inside one app, Postgres is fine. If the need is service isolation, replay, or independent scaling, evaluate dedicated messaging systems first.

      Attribution:
    • mjfisher #1
  3. 03

    Deletion strategy drives queue design

    The exchange around PgQue exposed that queue semantics follow from how you clean up processed work. Truncation-based designs can dodge bloat and locking costs, but they behave more like a log system and can be a poor fit for long-running jobs. Partitioning by time or type was suggested as a middle ground because it gives you coarse-grained cleanup without forcing one global truncate point.

    Choose cleanup mechanics before you choose a queue library. Long-lived jobs, retries, and per-job visibility rules can rule out the fastest table-reset approaches.

      Attribution:
    • KraftyOne #1
    • danielheath #1

Against the grain

  1. 01

    OpenAI is weak evidence for write-heavy queues

    Using OpenAI's PostgreSQL scaling post as proof that Postgres handles queues cleanly misses the workload details. The pushback was that OpenAI kept mostly read-heavy paths on a single primary with many replicas and moved harder write-heavy cases elsewhere, which is exactly where queue workloads often hurt most.

    Do not generalize from famous scale posts without matching workload shape. A read-heavy transactional system and a hot queue table stress Postgres in very different ways.

      Attribution:
    • tomnipotent #1 #2 #3
  2. 02

    The pattern is mature, not novel

    Some reactions were openly tired of yet another Postgres queue article built around `SKIP LOCKED`. That skepticism is useful because it reframes the post as operational guidance on a known technique, not a new discovery, and lowers the odds that a team mistakes familiarity for a solved production design.

    Treat this space as implementation detail, not frontier research. Compare operational tradeoffs and failure modes instead of being persuaded by the existence of another explainer post.

      Attribution:
    • richwater #1
    • tonyhb #1

In plain english

autovacuum ↩
PostgreSQL's background process that reclaims storage from dead tuples and updates statistics used by the query planner.
dead tuples ↩
Old row versions left behind by updates or deletes in PostgreSQL that still occupy space until vacuum cleanup removes them.
FOR UPDATE SKIP LOCKED ↩
A PostgreSQL query pattern that lets multiple workers select rows for processing without blocking on rows another worker already locked.
Kafka ↩
Apache Kafka, a distributed event streaming and log-based messaging system.
MVCC ↩
Multi-Version Concurrency Control, the PostgreSQL mechanism that keeps old row versions around so readers and writers can work concurrently.
Oban ↩
A PostgreSQL-backed job processing system for Elixir applications.
PgQue ↩
A PostgreSQL queue project designed to avoid table bloat by using paired tables and truncation-based cleanup.
Postgres ↩
PostgreSQL, an open source relational database system.
RabbitMQ ↩
A message broker used to route and deliver messages between systems.
Redis ↩
An in-memory data store often used for caching, pub-sub, and job queue backends.
SolidQueue ↩
A database-backed job queue adapter used with Ruby on Rails ActiveJob.
TRUNCATE ↩
A database operation that quickly removes all rows from a table without deleting them one by one.

Reference links

Queue libraries and implementations

  • PgQue
    Referenced as a PostgreSQL queue design that avoids tuple deletion and table bloat through table flipping and truncation.
  • Oban article: One Million Jobs a Minute with Oban
    Cited as evidence that PostgreSQL-backed job systems can sustain very high throughput in production-oriented benchmarks.

Prior discussion and scale references