HN Debrief

Kubernetes on Oxide: How customer needs shaped our integrations

  • Infrastructure
  • Open Source
  • Cloud
  • Developer Tools

Oxide’s post explains how customers pushed it to add a fuller Kubernetes story on top of its integrated on-prem cloud platform. The current approach is VM-first. Clusters are created with Cluster API through a provider called CAPOx, node lifecycle is handled by an out-of-tree cloud controller manager, and stateful workloads are being bridged for now with tools like Longhorn until Oxide ships its own Container Storage Interface plugin. Networking is also still in transition. Customers can already use external subnets to avoid overlay networking, and Oxide is using its cloud controller manager for Kubernetes LoadBalancer services, but tighter integration and a more complete load-balancing path are still being built.

If you are evaluating private-cloud infrastructure, read this as a signal that Oxide is building the Kubernetes pieces enterprise teams actually need, but key platform features like first-party storage and deeper networking integration are still arriving. For teams building similar platforms, the interesting move is not "Kubernetes support" by itself but using Kubernetes as a control surface on top of opinionated rack hardware and APIs.

Discussion mood

Positive and engaged. People mostly saw this as Oxide making real progress toward a credible Kubernetes platform, with excitement around Cluster API, networking, and future autoscaling. The main friction was that several pieces still feel incomplete, especially first-party storage, load balancing, and a clearer explanation of what Oxide actually is for outsiders.

Key insights

  1. 01

    Karpenter fits Oxide unusually well

    Oxide’s hardware model makes Karpenter more interesting than on clouds built around fixed instance catalogs. Because CPU and memory are tunable, a provider can synthesize many node shapes and place pods on much tighter-fit machines. That turns autoscaling from simple node count management into a packing advantage. It also exposes a real product boundary problem, since Cluster API and Karpenter both touch node lifecycle from different angles.

    If you are building on programmable infrastructure, autoscaling logic should exploit hardware flexibility instead of pretending every node is a pre-baked SKU. If you run both Cluster API and Karpenter, define ownership early or you will end up with two control loops fighting over nodes.

      Attribution:
    • bmwagner10 #1
    • sudomateo #1
  2. 02

    Oxide is choosing VM-first on purpose

    The VM layer is not a temporary compatibility shim. Oxide said its core primitive is the instance, backed by its own hypervisor and a full stack that includes hardware, firmware, operating system, and APIs. That makes "why not just run bare-metal Kubernetes" the wrong question. The product is meant to expose a managed virtualized substrate, not to compete with plain server vendors.

    Evaluate Oxide against private-cloud and virtualization platforms, not against bare-metal Kubernetes distributions. If your goal is direct host control above all else, this is a different product category.

      Attribution:
    • sudomateo #1
    • p_l #1
    • esseph #1
  3. 03

    Network integration is the differentiator to watch

    The most promising technical edge is not generic Kubernetes support. It is the chance to make pod and service networking line up with rack networking so clusters can use routed addresses and avoid overlays. Oxide staff said customers already use external subnets this way, and current Gateway API and LoadBalancer support works, but the deeper controller integration is still ahead.

    Watch whether Oxide turns its network model into simpler Kubernetes operations, not just feature parity checkboxes. If it succeeds, that could reduce overlay complexity, improve performance, and make on-prem clusters feel less like cloud emulations.

      Attribution:
    • sudomateo #1 #2
  4. 04

    Oxide already open sourced its RFD system

    The request to open source Oxide’s documentation workflow was answered with an existing public project, rfd-site, plus links to the internal RFD process it powers. That matters because Oxide’s public engineering culture is not just marketing polish. Some of the internal machinery behind how it writes and reviews technical decisions is already available.

    If your team likes Oxide’s engineering process, you can adopt parts of it now instead of treating it as vibe. The RFD tooling is a concrete artifact, not just a blog aesthetic.

      Attribution:
    • bcantrill #1

Against the grain

  1. 01

    The website still fails basic product explanation

    For someone coming in cold, the marketing copy leaves too much unanswered about what Oxide actually sells. The complaint was not about missing polish. It was about missing fundamentals like whether this is x86 hardware, where GPUs fit, what operating system is involved, and why Kubernetes sits on top of a non-Linux host. That confusion undercuts otherwise strong technical writing.

    If you sell deep infrastructure, make the entry path brutally concrete. Put the hardware model, virtualization model, and operating system story where a CTO can find them in minutes.

      Attribution:
    • ninkendo #1
  2. 02

    Home-lab demand is not a real market

    A small burst of enthusiasm for a hypothetical home version was met with a hard reality check. Oxide replied that it is targeting enterprises with huge public-cloud bills, and others pointed out that hobbyist admiration rarely converts into enterprise-scale hardware revenue. The gap between "I want one" and "I will sign a six-figure purchase order" is the whole business.

    Do not confuse audience excitement with addressable demand. If you are building expensive infrastructure, optimize for the buyer with a budget and a painful existing bill, not for fans who love the brand.

      Attribution:
    • lkasjdas #1
    • bcantrill #1
    • orf #1
    • asa400 #1

In plain english

CAPOx ↩
Oxide’s Cluster API provider, which lets Cluster API create and manage Kubernetes clusters on Oxide infrastructure.
cloud controller manager ↩
A Kubernetes component that integrates a cluster with infrastructure services such as node management, routes, and load balancers.
Cluster API ↩
An open source Kubernetes project for creating and managing Kubernetes clusters using Kubernetes-style resources and controllers.
Container Storage Interface ↩
A standard API that lets storage vendors integrate persistent storage with container orchestrators like Kubernetes.
Gateway API ↩
A newer Kubernetes API for defining network traffic routing and load-balancing behavior, intended to improve on older ingress patterns.
Karpenter ↩
An open source Kubernetes autoscaler that launches and removes machines based on the needs of pending workloads.
Kubernetes ↩
An open source system for deploying and managing containerized applications across clusters of machines.
KVM ↩
Keyboard, Video, Mouse switch, a device that lets one set of peripherals control multiple computers.
Longhorn ↩
An open source distributed block storage system commonly used to provide persistent volumes for Kubernetes workloads.
out-of-tree ↩
Built as a separate external component instead of being included directly inside the main Kubernetes codebase.
Proxmox ↩
A virtualization platform for managing virtual machines and containers on server hardware.
qemu ↩
An open source machine emulator and virtualizer often used to run virtual machines on Linux.
RFD ↩
Request for Discussion, a document format Oxide uses to propose and review technical decisions.

Reference links

Oxide engineering process and tooling

  • RFD 1
    Example of Oxide’s Request for Discussion process referenced when discussing its documentation system.
  • RFDs: The Backbone of Oxide
    Podcast episode explaining the RFD process behind Oxide’s engineering documentation.
  • rfd-site
    Open source repository for the documentation site system people were asking Oxide to release.

Kubernetes infrastructure integrations

  • Kubernetes cloudprovider interface
    Reference for the core interface an out-of-tree cloud controller manager must implement.
  • cloud-provider-aws
    Named as a current example of an out-of-tree cloud provider integration for Kubernetes.
  • Propolis
    Oxide’s hypervisor project, linked when explaining that Oxide does not use KVM or QEMU.

Home lab and adjacent platform references

Storage performance reference