HN Debrief

Solid Queue 1.6.0 now supports fiber workers

  • Programming
  • Developer Tools
  • Infrastructure
  • Open Source

Solid Queue is the database-backed background job system that Rails has been pushing as a simpler default, and version 1.6.0 adds fiber workers. In plain terms, that means a worker can juggle many paused and resumed I/O-bound jobs inside a small number of OS threads, instead of dedicating a thread to every job. That is a good fit for queues full of HTTP calls, fan-out jobs, notifications, and other work that spends most of its life waiting on networks rather than burning CPU.

If your Rails app does a lot of network-heavy background work, this makes Solid Queue worth revisiting before reaching for Sidekiq or a separate queue stack. Test your gems and database usage under fiber scheduling first, because the gains disappear fast when jobs are CPU-heavy or a dependency blocks the whole worker.

Discussion mood

Mostly positive and pragmatic. People liked seeing Rails close the gap on efficient I/O concurrency and liked that Solid Queue stays inside the Rails and Postgres stack, but they kept drawing a hard line around where fibers help and where they do not.

Key insights

  1. 01

    Separate worker pools are the real feature

    Separate queue configurations let you use fibers where they pay off and avoid them where they do not. The linked production config runs fiber-heavy pools for chat, Turbo, and notifications, then keeps a plain threaded pool for CPU work. That framing is more useful than asking whether fibers are "better" in the abstract, because most apps have both kinds of workloads.

    Split your background jobs by workload before you enable this. Put network-bound queues on fiber workers and isolate CPU-heavy queues so they do not clog the same execution model.

      Attribution:
    • pqdbr #1
    • QGQBGdeZREunxLe #1
  2. 02

    Durable fan-out jobs get much nicer

    Fan-out workflows that spawn many HTTP calls are where this lands immediately. Jobs already give you retries, persistence, and operational visibility. Adding fibers means you can keep that durability model and still run lots of waiting tasks efficiently, instead of building a separate async system just to avoid thread cost.

    If you are using background jobs as a reliability wrapper around external APIs, try replacing wide thread counts with fiber workers first. You may get the throughput bump you wanted without changing your retry and durability model.

      Attribution:
    • mrinterweb #1
  3. 03

    Database checkout behavior is the gating detail

    The important question is not raw fiber count. It is whether Rails holds a database connection across slow waits like outbound HTTP. One commenter noted that Rails now checks connections out and returns them around framework-controlled actions such as save calls, rather than pinning one for the whole worker flow. That change is what makes higher-concurrency fiber workers believable instead of a pool exhaustion trap.

    Profile connection checkout time, not just job concurrency. If your jobs keep a connection open while waiting on remote services, fix that first or fiber workers will only move the bottleneck to Postgres.

      Attribution:
    • resonious #1
    • achernik #1
  4. 04

    Fibers trade efficiency for weaker scheduling guarantees

    User-space scheduling is cheaper because the runtime knows what state to save and can switch work without a full OS thread handoff. The cost is that fairness gets worse. A badly behaved fiber can monopolize progress until it hits an I/O wait or some explicit yield point. That is a real operational constraint, not an academic footnote.

    Watch for long stretches of pure Ruby or native code inside fiber jobs. If work does not yield naturally, add boundaries or move it to a non-fiber pool so one hot job cannot stall many others.

      Attribution:
    • cogman10 #1

Against the grain

  1. 01

    Connection-pool advantage may be overstated

    Database connection savings are not unique to fibers. Threads also do not need one connection each if handlers borrow from a pool briefly and block when the pool is empty. That undercuts any sales pitch that treats lower connection counts as automatic proof that fibers are the better design.

    Do not choose fibers on the assumption that they solve connection scaling by themselves. Compare actual checkout patterns in your app, because a well-behaved threaded worker pool may already use the database efficiently.

      Attribution:
    • alex_smart #1
  2. 02

    Ruby versus Go benchmark claims need context

    Near-parity latency anecdotes for HTTP tests say more about a narrow benchmark and native code in the web stack than about Ruby broadly catching Go. Without concurrency level, machine details, percentile spread, and workload composition, the result is interesting but not decision-grade evidence.

    Treat language-performance claims here as a prompt to benchmark your own workload, not as a green light to generalize. Include memory, concurrency, and where time is spent in native code before you draw architecture conclusions.

      Attribution:
    • jerf #1
    • asa400 #1

In plain english

C extension ↩
A library component written in C that a higher-level language like Ruby calls for speed or system access.
CPU-bound ↩
Work whose speed is limited mainly by processor time rather than waiting on external systems.
fiber ↩
A lightweight unit of concurrent execution managed by the language runtime, which can pause and resume work without creating a full operating system thread.
Go ↩
A compiled programming language from Google known for fast execution and built-in concurrency support with goroutines.
I/O ↩
Input and output operations such as network calls or disk access, where a program often spends time waiting for external systems.
OS thread ↩
A thread scheduled directly by the operating system, with more overhead than a runtime-managed fiber.
Redis ↩
An in-memory data store commonly used as a cache, message broker, or job queue backend.
Turbo ↩
A Rails front-end technology for partial page updates and real-time interactions without full page reloads.

Reference links

Solid Queue and async Ruby context

Prior art and related runtimes

  • EventMachine
    Raised as earlier Ruby prior art for event-driven and asynchronous I/O.
  • BEAM (Erlang virtual machine)
    Linked to explain the Erlang and Elixir runtime that commenters compared favorably against Ruby's concurrency model.

Benchmarks and implementation details