HN Debrief

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

  • AI
  • Security
  • Developer Tools
  • Open Source
  • Infrastructure

The post analyzes data from a browser game that mimics a common coding-agent workflow: the agent asks to run commands, the user has little time, and some commands hide dangerous behavior in places like shell history or npm scripts. After about 40,000 plays and more than 400,000 decisions, the author says players missed about 1 in 3 threats, with npm-related prompts and long, noisy command lines doing especially well at slipping through. The article frames this as evidence that human-in-the-loop approval is brittle once people are rushed, fatigued, or trusting the tool.

Do not treat click-to-approve flows as your primary defense for agentic coding tools. If you use agents in real work, push the risk down into sandboxing, scoped capabilities, network limits, and automated policy checks before these tools get deeper access to code, credentials, or production systems.

Discussion mood

Mostly skeptical of permission prompts and broadly supportive of stronger containment. The mood was less outrage at the article than resignation that this is the latest version of a long-failed security pattern, with added concern that vendors use prompts to offload responsibility while chasing more autonomous features.

Key insights

  1. 01

    Useful agent security is harder than it sounds

    The hard part is not locking an agent down completely. It is defining what a secure but still useful agent even is. Once an agent can read local files, talk to the web, or combine multiple data sources, it can create exfiltration paths and side channels that simple allow or deny prompts will never model well. Comparing this to securing a powerful human operator is the right frame. You are not approving isolated commands. You are governing an adaptive actor with broad, composable actions.

    If you are designing agent controls, start by writing down the exact capabilities and data flows you will permit. If you cannot state those clearly, you do not yet have a real security model and should default to much tighter isolation.

      Attribution:
    • jerf #1 #2
    • cogman10 #1
    • 20k #1
    • friendzis #1
  2. 02

    Teams already prefer containment over prompts

    People with hands-on setups described a consistent operating model. Run the agent as an unprivileged user or in a VM or container, block access to home directories and config files, gate outbound network paths, keep secrets out of ambient environment variables, and have the agent produce diffs or pull requests instead of writing directly into the trusted workspace. The point is not perfect security. It is shrinking blast radius enough that turning off endless prompts becomes sane.

    Treat your coding agent like untrusted contractor software on a throwaway machine. Give it a scoped workspace, no ambient credentials, and a reviewable output path such as patches or pull requests.

      Attribution:
    • Tuna-Fish #1
    • danudey #1
    • gopher_space #1
    • kstenerud #1
  3. 03

    Auto-approvers help, but only within narrow policy

    Several commenters said the next obvious layer is a second model or policy engine that reviews the first model’s actions. That can cut down on approval fatigue and catch routine bad calls. The catch is that these systems are only as good as the policy they enforce. They struggle with intent mistakes, dangerous command flags, or context-dependent tool usage unless the harness is expressive enough to describe those cases and escalate uncertainty back to a human.

    Use automated approval to filter routine actions, not as a blanket substitute for controls. Make sure your harness can express fine-grained rules and route ambiguous cases to review instead of forcing binary auto-approve or auto-reject.

      Attribution:
    • Aurornis #1 #2
    • Tuna-Fish #1
    • dgunay #1
    • hombre_fatal #1
  4. 04

    Prompts answer the wrong question

    The sharpest criticism was not that people click too fast. It was that the prompt appears at the moment when the user has the least information. A shell command string asks the human to predict future consequences from partial context. What you can actually verify is the resulting artifact, diff, deployment, or network effect after execution. That makes raw command approval both noisy and badly placed as a control surface.

    Move review up a level from shell text to outcomes. Inspect patches, filesystem accesses, network intents, and produced artifacts rather than expecting people to reason about every inline script or chained command.

      Attribution:
    • IgorVoytyuk #1
    • whazor #1
    • cmiles8 #1
    • continuational #1
  5. 05

    Human-in-the-loop doubles as a liability shield

    A recurring view was that approval prompts persist because they assign responsibility, not because they work well. By requiring a human click, vendors can say the user accepted the action and absorb less blame when the agent does something harmful. The phrase “moral crumple zone” came up for exactly this pattern. Automation gets the upside while the human operator gets the accountability when it fails.

    When you evaluate agent products, read permission workflows as legal and organizational design as much as technical design. Ask who is expected to absorb the failure when the model makes a bad decision, then engineer your internal controls accordingly.

      Attribution:
    • grndn #1
    • anal_reactor #1
    • stingraycharles #1
    • est31 #1

Against the grain

  1. 01

    The game is too weak to support strong claims

    Several people argued the experiment is closer to a provocative demo than a meaningful benchmark. The timer creates artificial pressure, the scoring encourages speed over caution, some npm and shell examples depend on hidden prior context, and a few dangerous or safe labels look inconsistent. On that reading, the 1-in-3 number says more about this game design than about real approval behavior in a serious engineering environment.

    Do not import the exact miss rate into your risk model. Use this as a warning about alert fatigue, then validate your own workflows with realistic tasks, stakes, and policy definitions.

      Attribution:
    • VladVladikoff #1
    • Kinrany #1
    • Surac #1
    • stonedivot #1
    • tonymet #1
    • Aurornis #1
  2. 02

    Security limits can destroy the product value

    One pushback was that strict controls often collapse the utility that makes coding agents attractive in the first place. If every useful path requires exposing more files, more tools, or broader system access, people will punch holes through the sandbox until convenience wins. From that view, safe but crippled software loses to unsafe but highly productive software in real organizations.

    Expect users to route around controls that block common work. Security designs for agents need to preserve a fast path for legitimate tasks or they will be bypassed in practice.

      Attribution:
    • applfanboysbgon #1 #2
    • tcdent #1
    • mafuy #1
  3. 03

    File-centric permissions are only part of it

    Some wanted permissioning to focus on file reads and writes rather than shell commands, because command text is too long and too opaque to review. Others pushed back that this still misses process execution, network effects, and side channels. Existing systems like SELinux and AppArmor already show that no single dimension of control is enough. Files, commands, and network all matter.

    Do not replace command prompts with a single new primitive and call it done. Agent controls need multiple layers that cover filesystem scope, process execution, and outbound communication together.

      Attribution:
    • wmanley #1
    • crabbone #1

In plain english

AppArmor ↩
A Linux security module that restricts programs according to per-application security profiles.
diff ↩
A structured view of what changed between two versions of files or code.
exfiltration ↩
The unauthorized transfer of sensitive data out of a system.
human-in-the-loop ↩
A system design where a person must review or approve actions before the software continues.
npm ↩
Node Package Manager, the standard package manager and script runner used in JavaScript and Node.js projects.
sandbox ↩
An isolated execution environment that limits what a program can access or change on the host system.
SELinux ↩
Security-Enhanced Linux, a Linux security framework for mandatory access control policies.
VM ↩
Virtual machine, a software emulation of a computer used to isolate workloads from the main system.

Reference links

Game and article

Security concepts and references

Languages and capability systems

Tools and products

  • formwork sandbox project
    Shared as a generic sandbox environment for locking down files, network, and tools from agents.
  • yoloAI
    Presented as an open source agent runner built around sandboxing instead of permission prompts.
  • Tigera Lynx
    Mentioned as a product for policy-driven control of scoped agents, networks, and tool access in organizations.
  • rik
    Linked by a commenter discussing self-approval behavior in agent harnesses they built.

Documentation and standards