HN Debrief

DskDitto: Ultra-fast, parallel duplicate-file detector

  • Open Source
  • Storage
  • Developer Tools
  • Programming

DskDitto is a Go-based duplicate-file finder that promises very fast scanning across large disks and includes features like fuzzy matching. People quickly treated it as one more entrant in a crowded category rather than a breakthrough, and the useful conversation centered on implementation choices. The biggest technical pushback was that duplicate detection does not need cryptographic hashing for the first pass. Fast non-cryptographic hashes or even CRC-32 can cheaply eliminate almost all non-matches, with slower detailed checks only on collisions. That fits what several people see in practice anyway, because these tools often hit storage and memory bandwidth limits long before they run out of CPU. Comments also surfaced adjacent features users actually want from dedupe software: whole-directory duplicate detection, media-aware perceptual matching like Czkawka offers, and reflink conversion so duplicate files become copy-on-write clones instead of getting deleted. On usability, the strongest praise went to tools like fclones that produce intermediate candidate lists for review before any destructive action. The mood was interested but grounded. People liked that the author shipped something useful, but they judged it against existing tools and against the real bottlenecks of disk scanning rather than the repo’s “instantly” framing.

If you build or adopt dedupe tooling, pay more attention to hashing strategy, I/O bottlenecks, and safe review workflows than headline benchmark claims. The winning products here are the ones that combine fast candidate filtering with conservative deletion or reflink workflows users can trust.

Discussion mood

Mildly positive and practical. People were glad to see another useful filesystem tool, but most reactions were skeptical of the speed claims and focused on real-world concerns like hash choice, I/O limits, safe workflows, and comparison with established alternatives.

Key insights

  1. 01

    Use cheap hashes as a front filter

    Using a cryptographic hash for every file is wasted work when the job is simple duplicate detection. A fast hash like xxHash or even CRC-32 can reject almost all files quickly, then a second table or deeper check can handle the tiny set of possible collisions. That design changes the performance profile from compute-heavy to selective verification.

    If you are building internal dedupe or backup indexing, separate candidate generation from confirmation. Store a cheap fingerprint in memory first, then pay for stronger checks only on likely matches.

      Attribution:
    • divyekapoor #1
    • LorenPechtel #1
  2. 02

    Disk I O is the real ceiling

    Parallel hashing sounds CPU-bound until you run it on real machines. People reported saturating disk-to-memory throughput well before maxing out cores, and the practical tuning knob was worker count rather than brute-force parallelism. The comments also noted OS-specific limits around open file handles, which makes aggressive concurrency less portable than it looks in a benchmark.

    Benchmark dedupe tools against your storage stack, not your CPU. Expose concurrency controls and test on Linux, macOS, and Windows separately if you expect the same codepath to scale across all three.

      Attribution:
    • lanstin #1 #2
    • indrora #1
  3. 03

    Reflinks are better than deletion

    Reflink conversion was proposed as a higher-value outcome than simply identifying duplicates. On filesystems that support copy-on-write cloning, you can collapse identical files to shared storage blocks while preserving normal file behavior. That gives users disk savings without the operational risk of deleting something they still expect to exist as a separate file.

    If your product targets developer workstations, media libraries, or backups, consider reflink support as a first-class action. It delivers dedupe savings with a much safer user experience than automatic removal.

      Attribution:
    • Maakuth #1
  4. 04

    Reviewable candidate lists build trust

    The appeal of fclones was not just raw performance. Its workflow produces intermediate files of duplicate candidates so users can inspect and script decisions before touching data. That extra step makes a dedupe tool feel safer and more automatable, especially when the cost of a false positive is high.

    Treat output format and review flow as part of the core product. Teams will adopt a slightly slower tool if it gives them an auditable path before destructive changes.

      Attribution:
    • jauntywundrkind #1
  5. 05

    Exact duplicates and fuzzy matches are separate jobs

    Image and video libraries push duplicate detection beyond byte-for-byte equality. Czkawka was cited because it supports perceptual hashes for media, while DskDitto’s current fuzzy mode uses SimHash and drew questions about how robust that is. That highlights a real product split between exact dedupe and similarity search.

    Be explicit about whether your tool finds identical files or similar content. If you support fuzzy matching, document the algorithm and expected false positives so users know when they can trust the result.

      Attribution:
    • psYchotic #1
    • jdefr89 #1
    • akoboldfrying #1

Against the grain

  1. 01

    Instantly is marketing not reality

    The joke about finding duplicates across large disks "instantly" landed because everyone knows disk scans are bounded by hardware. Even well-written tools still have to read metadata and often file contents, so the speed claim reads as repo hype rather than an engineering statement.

    Be careful with performance language in developer tools. Concrete benchmark conditions will earn more trust than broad speed promises.

      Attribution:
    • bronlund #1

In plain english

copy-on-write ↩
A storage strategy where data is shared until one copy is modified, at which point only the changed parts are written separately.
CRC-32 ↩
Cyclic Redundancy Check 32-bit, a fast checksum used to detect accidental data changes but not designed for security.
reflink ↩
A filesystem feature that creates copy-on-write clones so multiple files can share the same underlying data blocks.
SimHash ↩
A hashing technique that maps similar inputs to similar fingerprints, often used for approximate or fuzzy matching.
xxHash ↩
A very fast non-cryptographic hash function often used for checksums and data indexing.

Reference links

Alternative dedupe tools

  • Czkawka
    Referenced as a similar tool that also supports perceptual hashing for images and videos.
  • fclones benchmarks
    Referenced as the main competing tool to compare against, especially on performance and workflow.

Related hashing and parallel scan projects

  • hyperhash
    Shared as a Go project exploring massive parallel hashing and practical OS limits.
  • libcircle
    Mentioned as a possible point of comparison for parallel traversal or distributed work scheduling.

Implementation references