HN Debrief

Water system controllers don't belong on the internet, says ex-NSA chief

  • Security
  • Infrastructure
  • Hardware
  • Regulation

The article says water utility control systems should not sit on the open internet, especially after recent suspected attacks, and uses that to argue for more isolation around operational technology. The comments mostly landed on a narrower and more practical point: the real failure is not merely "internet connected" versus "offline" but that industrial control environments are still built around ancient operating systems, flat networks, unsigned vendor tools, weak physical security, and teams that were never given modern software or security practices.

If your company touches facilities, labs, utilities, or industrial gear, assume ordinary IT controls and software practices do not exist by default. Treat internet exposure, vendor tooling, and remote access paths as board-level risk, not an ops footnote.

Discussion mood

Strongly alarmed and cynical. Most comments agreed that public internet exposure for water controls is reckless, and the mood darkened because practitioners said the underlying operational technology stack is far older, less testable, and less governable than outsiders assume.

Key insights

  1. 01

    Industrial controls still run on pre-DevOps habits

    In practice this world often looks less like modern software and more like ad hoc file handling on Windows laptops, unsigned vendor executables, and firmware workflows with no version control, test harnesses, or safe rollback path. That changes the risk model. A mistake is not a bad deploy you can revert in minutes. It can be the wrong customer program pushed into a live plant or a field visit blocked because the only tool is an admin-only binary from a Taiwanese FTP server.

    If you oversee any operational technology estate, audit the engineering workflow itself, not just the network diagram. Source control, golden images, controlled toolchains, and tested recovery paths will reduce risk faster than another policy memo about cyber hygiene.

      Attribution:
    • aliasxneo #1
    • technion #1
    • steve_adams_86 #1
    • b40d-48b2-979e #1
  2. 02

    Controls work demands domain knowledge most software teams lack

    The bottleneck is not just bad culture. PLC work sits at the intersection of real-time computing and physical process engineering, where timing guarantees, noisy industrial I/O, and concepts like head pressure or pump cavitation matter as much as code quality. That is why a strong software engineer can still do expensive damage, and why a Raspberry Pi is not a substitute for a certified rugged deterministic controller.

    Do not assume you can rotate generalist app engineers into industrial automation and get the same results. Hire for controls experience or pair software modernization efforts with people who understand the plant physics and failure modes.

      Attribution:
    • Kim_Bruning #1
    • aliasxneo #1
    • amelius #1
    • stackghost #1
    • tamimio #1
  3. 03

    The real design question is remote access architecture

    What matters is not a binary "internet or no internet" label but how remote access is structured for widely distributed assets. Water towers, lift stations, and flood control equipment often need remote operation, so the defensible pattern is mediated access through hardened endpoints, separate management machines, and network segmentation that limits what happens if connectivity is lost or an operator laptop is compromised. A tunnel over shared infrastructure may even beat single-path private fiber on resilience, provided the trust boundary is tight.

    Review remote access as a system design problem with failure modes, not a checkbox. Ask which devices can initiate access, what happens during a denial-of-service event, and whether operations can fail safely when the control path disappears.

      Attribution:
    • Terr_ #1
    • lokar #1 #2 #3
    • baby_souffle #1
  4. 04

    Air gaps do not cover radio and field devices

    Many systems that are not exposed on the public internet still depend on insecure RF links, unencrypted serial protocols, and remote terminal units sitting in lightly protected roadside enclosures. Radio leaks beyond the fence, field cabinets are often easy to access, and many SCADA deployments implicitly trust anything that speaks like a legitimate endpoint. That means "offline" systems can still be reachable enough for a determined attacker.

    Include wireless links, serial protocols, and physical enclosures in your threat model. If your security review stops at public IPs and firewalls, you are missing some of the easiest paths into operational networks.

      Attribution:
    • clbrmbr #1
    • lll-o-lll #1
    • procarch2019 #1
    • aliasxneo #1
  5. 05

    Maintenance economics keep beating security advice

    The hard truth is that these systems stay fragile because proactive maintenance is politically and financially disfavored. Operators are rewarded for avoiding downtime and staying within budget, not for replacing an old but functioning control stack before it fails or gets hacked. Without enforceable standards and inspections, network security stays a vague best practice instead of a shutdown-worthy compliance requirement like a structural defect in a building.

    If you want better security outcomes, budget and governance have to force modernization before an incident. Internal audits and insurer or regulator requirements are more likely to move behavior than another round of voluntary guidance.

      Attribution:
    • Jtsummers #1
    • 27183 #1 #2
    • Kim_Bruning #1

Against the grain

  1. 01

    Some operators prefer one-way visibility over control

    For the highest-risk systems, remote monitoring without remote actuation can be the right compromise. The webcam-pointing-at-gauge example sounds primitive, but it preserves visibility while eliminating the command path that turns convenience into sabotage. In this framing, the safest internet connection is one that cannot issue control inputs at all.

    Separate observation from control wherever you can. If a site only needs status data, use read-only channels or out-of-band sensing instead of extending the plant control network.

      Attribution:
    • 1970-01-01 #1
    • eek2121 #1
    • snypher #1
  2. 02

    Government should do active defense at scale

    A smaller but forceful view was that guidance is nowhere near enough when foreign state actors are probing civilian infrastructure. It argues national security agencies should use modern coding agents and offensive access to infiltrate hostile hacking teams, disrupt campaigns early, and treat utility defense as part of foreign intelligence work rather than leaving thousands of local operators to fend for themselves.

    If your threat model includes nation states, do not assume local hardening is the whole answer. Watch for expanded public-private defense programs, because the operational burden may shift if governments move from advisory roles to direct disruption.

      Attribution:
    • jacobgold #1 #2

In plain english

FTP ↩
File Transfer Protocol, an old method for downloading and uploading files over a network that is often insecure by modern standards.
I/O ↩
Input Output, the signals a system receives from sensors and sends to equipment like motors or valves.
PLC ↩
Programmable Logic Controller, a rugged industrial computer used to control machines and physical processes in factories, utilities, and buildings.
RF ↩
Radio Frequency, wireless communication carried over radio waves.
SCADA ↩
Supervisory Control and Data Acquisition, a class of systems used to monitor and control industrial sites such as water plants, power systems, and pipelines.

Reference links

Government security guidance

Industrial control case studies and examples