HN Debrief

Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

  • Infrastructure
  • Open Source
  • Linux
  • Storage
  • Developer Tools

The post is a systemd issue report showing that systemd-journald can turn a single short log entry into 49 KB of writes on ext4 and more than 110 KB on btrfs. The author is not just complaining about disk usage in the abstract. They measured physical writes and tied the effect to frequent small updates, which become especially painful on copy-on-write filesystems where tiny random writes are expensive. That landed because plenty of people have seen the same symptom from the outside. An otherwise idle desktop keeps burning through SSD writes, and when they go looking, journald is often part of the bill.

If you run Linux systems on SSDs, laptops, or btrfs, treat local logging as a storage design choice, not a default you never revisit. Check your idle write rate, consider volatile journald storage or forwarding to another logger, and audit chatty apps because journald magnifies their damage.

Discussion mood

Strongly negative. Most comments treat the measurements as confirmation of long-standing frustrations with journald’s performance, file format, and poor controls for noisy log sources. Even people who defend structured querying mostly concede that the write path looks badly matched to a logger’s workload.

Key insights

  1. 01

    Format churn looks like the root cause

    The likely culprit is not logging volume alone but how journald lays data out on disk. The format appears to scatter each entry across small updates to indexes and metadata inside the same file, so one new line can dirty whole blocks and pages. That turns a logger into a random-write workload, which is exactly what filesystems and SSDs handle least efficiently for tiny frequent writes.

    If you are investigating SSD wear or unexplained background writes, look past application log volume and inspect the logger’s storage pattern. An append-only sink or batched index updates will usually beat clever per-entry indexing for host-local logs.

      Attribution:
    • pengaru #1 #2
    • ValdikSS #1
  2. 02

    The real failure is poor flood control

    Once a service or driver starts spamming logs, journald gives you too few ways to stop the bleeding without blunt global tradeoffs. LogFilterPatterns exists, but people described it as service-scoped, awkward, and useless for kernel-originated noise. That makes routine incidents like a broken GPU resume path or a noisy desktop component much more expensive than they should be.

    Plan for log storms before you hit one. Document how to switch to volatile storage, where to add per-service filters, and which sources still need rsyslog or another downstream tool for real suppression.

      Attribution:
    • 0x_rs #1
    • jck86 #1
    • mzajc #1
  3. 03

    Existing databases do not make this easy

    The easy fix is not "just use SQLite" or "just use DuckDB". Commenters pointed out that durable small writes, concurrent readers, compact storage, and open tooling pull in different directions. SQLite is robust and widely deployed, but it is still a rewritable database. DuckDB and Parquet look attractive for compression and analytics, but they are awkward for low-latency durable appends. The useful point is narrower: journald’s current design is bad, but replacing it cleanly is a storage-engine problem, not a weekend refactor.

    If you are building structured local logs, write down your durability, read concurrency, and append behavior first. Picking a familiar database without matching it to the workload can just move the problem around.

      Attribution:
    • quotemstr #1
    • orf #1
    • ElectricalUnion #1
    • pengaru #1
    • amluto #1
  4. 04

    Local indexing lost its strategic value

    Host-local query features mattered more when machines were less ephemeral and off-host log pipelines were less universal. Today, serious search and indexing usually live in Loki, Elasticsearch, or Splunk, while the local logger mainly needs to be durable, cheap, and available during outages. That weakens the case for paying a heavy per-write tax on every machine just to support richer local queries.

    Separate the jobs of local logging and fleet-wide analytics in your architecture. Optimize the on-host logger for low overhead and failure isolation, then do indexing and search after logs leave the box.

      Attribution:
    • lokar #1
    • hedora #1
    • otterley #1
  5. 05

    Journald is only one part of the write budget

    The measurements resonated because many desktop and server components are already excessively chatty before journald amplifies their output. People called out Plasma components, browser telemetry stores, Bitwarden, IPFS, Docker networking churn, Redis snapshots, and kernel audit noise. Journald becomes the multiplier on top of a larger ecosystem problem of software that treats background disk writes as free.

    Do not stop at the logger when trimming SSD wear. Profile which applications write while idle, fix or replace the worst offenders, and expect the biggest gains from reducing events before they ever reach journald.

      Attribution:
    • ValdikSS #1
    • graemep #1
    • doublepg23 #1
    • marginalia_nu #1

Against the grain

  1. 01

    Field queries still beat grep

    Structured queries are not fake value just because ripgrep is fast. Querying by fields is still different from scanning text, and for some debugging workflows journalctl’s indexed metadata is genuinely useful. The complaint here is about the implementation cost, not the idea that local logs should carry structure.

    Do not throw away structured logging because this implementation is clumsy. Keep the fields, but be skeptical of designs that maintain expensive indexes on every write.

      Attribution:
    • altairprime #1
    • otterley #1
  2. 02

    Parquet is the wrong ingest layer

    Columnar file formats sound elegant until you remember that a logger must accept durable writes continuously. Parquet wants bigger batches and stores critical metadata in the footer, so readers cannot reliably treat an actively written file like a finished dataset. That makes it a poor direct replacement for a live system journal.

    Use analytics formats after compaction, not on the hot ingest path. If you want columnar storage, add it as a downstream format once logs are safely captured.

      Attribution:
    • orf #1

In plain english

btrfs
A Linux file system with features like snapshots and checksumming, often chosen for advanced storage management.
copy-on-write
A technique where data is shared until a change is made, at which point a modified copy is created instead of changing the original in place.
DuckDB
An open-source analytical database designed to run in-process, often embedded inside applications or notebooks, and optimized for fast SQL queries on tabular data.
Elasticsearch
A distributed search engine built on Lucene, commonly used for full-text search and log analysis.
ext4
A widely used Linux file system for storing data on disks.
LogFilterPatterns
A systemd service setting that can include or exclude log messages matching given text patterns.
Loki
Grafana's log aggregation system.
mmap
A system call that maps memory pages into a process address space, often used by allocators to obtain memory from the operating system.
page cache
The operating system’s in-memory cache of file data used to speed up disk access.
Parquet
A columnar data storage format commonly used in analytics systems.
rsyslog
A traditional syslog daemon for collecting, filtering, and forwarding logs on Unix-like systems.
Splunk
A commercial platform for collecting, searching, and analyzing machine-generated logs and telemetry.
SQLite
A lightweight relational database that runs from a local file and is often used for simple deployments or embedded apps.
SSD
Solid-State Drive, a storage device that uses flash memory instead of spinning disks.
systemd-journald
The logging service in systemd that collects and stores system and application logs on many Linux systems.
write amplification
A situation where a small logical write causes much more physical disk writing than expected.

Reference links

Issue and format documentation

systemd manuals and historical context

Alternative distributions and logging approaches

  • Devuan
    Suggested by people who want to avoid systemd entirely.

Related fixes and performance notes

Examples of other chatty software