HN Debrief

thunderbolt-ibverbs: We have InfiniBand at home

  • Hardware
  • Infrastructure
  • Open Source
  • AI

The post describes an experimental Linux driver, thunderbolt-ibverbs, that exposes a Thunderbolt connection as an InfiniBand verbs device. In plain terms, it lets software written for RDMA-style networking use a Thunderbolt cable as the transport, even though there is no actual InfiniBand fabric underneath. The author used it with higher-level GPU communication libraries like NCCL and JACCL, aiming at the common problem of wiring together a couple of machines or accelerators without paying for datacenter interconnect hardware.

If you are building small GPU clusters or workstation-to-workstation links, watch this space as a way to get much of RDMA’s software stack without buying dedicated InfiniBand hardware. Also track Linux USB4STREAM support, because the kernel is moving toward first-class raw Thunderbolt data paths that could make this kind of setup cleaner and less custom.

Discussion mood

Positive and curious. People liked the hack, liked the author being candid about the project’s rough edges, and focused on the practical upside of getting RDMA-style software working over common Thunderbolt hardware. The main skepticism was technical precision, not hostility: commenters wanted to separate raw Thunderbolt transport from ordinary Thunderbolt networking and to clarify that this is verbs emulation, not actual InfiniBand.

Key insights

  1. 01

    Linux is adding a cleaner raw Thunderbolt path

    Linux is already moving toward an official way to move data over raw Thunderbolt with USB4STREAM. That makes this project look less like a one-off stunt and more like an early user-space and driver experiment sitting in front of a kernel capability that is about to become standard. The interesting split is interface choice. USB4STREAM exposes a simple read and write file abstraction, while thunderbolt-ibverbs exposes an RDMA verbs device so existing GPU communication software can use it immediately.

    If this matters to your roadmap, test against upcoming Linux 7.2-era kernel support instead of treating the post as a dead-end prototype. Decide early whether your stack wants a generic byte stream or compatibility with libibverbs-based software.

      Attribution:
    • l1k #1
    • adrian_b #1
  2. 02

    The bandwidth gain came from using more rails

    The jump over thunderbolt-net did not come from vague “lower overhead” alone. The author says the stock networking path usually binds to one Thunderbolt rail and leaves the second pair, or even extra cables, unused. Their verbs-style transport could stripe across rails because it does not have to preserve the full semantics of an Ethernet and IP stack. That is why the standard Thunderbolt network path sat near 9 to 10 Gb/s on their setup while the custom path got far closer to the cable’s real capacity.

    Do not compare transports by cable label alone. For any direct-attach interconnect, ask whether the software stack can actually use all available lanes and links before you assume the physical spec is your bottleneck.

      Attribution:
    • grw_ #1 #2
  3. 03

    This is RDMA API compatibility, not InfiniBand fabric compatibility

    The project is best understood as implementing the libibverbs contract over Thunderbolt, not reproducing the InfiniBand network stack. That is why it can impersonate a RoCE v2 device even though there is no Ethernet beneath it. It also means “bridging” into real InfiniBand hardware is not some automatic property you get from the API. A real bridge would need hardware or software that translates semantics across very different link layers, and here that translation burden lands on the host CPU.

    If you depend on software that only needs verbs semantics, this approach may be enough. If you need interoperability with actual InfiniBand deployments or switch fabrics, treat that as a separate engineering project, not an implied feature.

      Attribution:
    • grw_ #1 #2
    • eqvinox #1
  4. 04

    Thunderbolt controller details still dominate performance

    The rough performance shape appears to depend heavily on controller implementation and low-level buffer mechanics, not just on the cable generation. The author saw very different one-way numbers across Strix Halo, another AMD chipset, and an Apple Thunderbolt 5 Mac. They suspect fixed 4 KB page handling in communication with the NHI may be part of the bottleneck on the weaker setup. That points to an uncomfortable truth. This is close enough to DMA plumbing that platform quirks can swamp elegant protocol ideas.

    Benchmark on the exact host platforms you plan to ship or deploy. For low-level interconnect work, chipset and controller behavior can matter as much as the nominal link speed on the box.

      Attribution:
    • grw_ #1
    • scottlamb #1
  5. 05

    Candor about AI-generated code helped credibility

    People responded well to the project being explicit that AI was used to write parts of the code and that breakage should be expected. That honesty lowered the usual alarm around AI-assisted systems code because readers could calibrate trust correctly. In a project this close to the kernel and DMA, that framing matters. It signals that the result is a useful experiment, not production-ready infrastructure pretending to be polished.

    If you publish AI-assisted low-level code, label it plainly and set expectations up front. You will get better feedback and less backlash than if readers discover the rough edges on their own.

      Attribution:
    • mkesper #1
    • kjs3 #1

Against the grain

  1. 01

    Used InfiniBand gear may still be the saner buy

    For a pair of Linux machines, cheap second-hand adapters and cables can get you actual InfiniBand with better performance and fewer caveats. That undercuts the romance of the hack if your main goal is stable throughput rather than experimentation. The author’s answer is pragmatic, not technical. Specialized gear still costs real money and time, while a Thunderbolt cable may already be sitting on your desk.

    Price out used Mellanox hardware before committing engineering time to custom transport work. If the gap is small, buying the real thing may be cheaper than debugging a clever substitute.

      Attribution:
    • speedbird #1
    • grw_ #1

In plain english

ASIC ↩
Application-Specific Integrated Circuit, a chip built for a particular job such as handling network protocols at very low latency.
DMA ↩
Digital Markets Act, a European Union law aimed at limiting anti-competitive behavior by large digital platform gatekeepers.
InfiniBand ↩
A high-performance networking standard used in servers and clusters, especially for low-latency data transfer between machines.
JACCL ↩
A GPU communication library mentioned in the comments that builds on RDMA-style communication patterns for accelerator workloads.
libibverbs ↩
A userspace library that exposes the standard verbs API used by InfiniBand and other RDMA-capable devices.
NCCL ↩
NVIDIA Collective Communications Library, a software library used to move data efficiently between GPUs for distributed training and other parallel workloads.
NHI ↩
Native Host Interface, the Thunderbolt controller interface used by the host system to manage low-level data movement.
RDMA ↩
Remote Direct Memory Access, a way for one computer or device to read or write another system’s memory with very low CPU involvement.
RoCE v2 ↩
RDMA over Converged Ethernet version 2, a way to provide RDMA-style networking over Ethernet and IP networks.
Strix Halo ↩
An AMD system architecture with large shared memory that some people use to experiment with running big models locally.
thunderbolt-net ↩
The Linux networking driver that exposes a Thunderbolt connection as a conventional IP network interface.
USB4STREAM ↩
A new Linux kernel feature for sending data over raw Thunderbolt or USB4 streams with minimal overhead.

Reference links

Kernel and driver references

Coverage