HN Debrief

Benchmarking Opus 5 on SlopCodeBench

  • AI
  • Developer Tools
  • Programming

The post benchmarked Opus 5 on a subset of SlopCodeBench, a research benchmark built to catch a failure that normal coding evals miss. Instead of grading one isolated task, it makes an agent add features checkpoint by checkpoint and then checks whether the resulting code still works and stays manageable. On that setup, Opus 5 posted only a small absolute gain over older Claude results from the paper, while also generating far more functions and generally exhibiting the kind of code bloat the benchmark is meant to expose.

If you use coding agents in production, start measuring what happens after the third and fourth change, not just whether the first task passed. Add review loops, complexity checks, and your own evals now, because model upgrades and prompt tweaks can easily trade short-term output for long-term codebase drag.

Discussion mood

Interested and mostly positive about the benchmark, with frustration aimed at coding agents that inflate complexity over time and at labs that seem to optimize for impressive first-pass demos instead of maintainable code. The mood is tempered by skepticism about benchmark methodology, sample size, and whether harness design matters more than the underlying model.

Key insights

  1. 01

    Static skill packs can hurt greenfield agents

    Preset workflow packs look less like accumulated wisdom and more like stale baggage on this benchmark. The key point is not just that some skills failed to help. They may actively reduce pass rates because they consume context and push investigation-heavy routines that make sense on large existing repos, but are wasted on greenfield tasks like SlopCodeBench. Newer models also seem better at choosing an ad hoc workflow for the problem in front of them than following a canned playbook.

    Re-test any agent skill bundle you ship instead of assuming more scaffolding helps. Keep separate configurations for greenfield work and existing codebases, because the same workflow can help one and drag down the other.

      Attribution:
    • gck1 #1
    • robbomacrae #1 #2
    • dhorthy #1
  2. 02

    Agents need machine-readable project state

    What helps agents stay on track is not a longer markdown plan. It is exposing state in a form they can parse and act on. The useful framing here is that a CLI can compress a messy project into structured visibility and valid transitions, much like a state machine. That reduces drift during long runs and survives context compaction better than free-form notes. Types were also proposed as a practical approximation of system state because models can read them directly and use them as guardrails.

    If your agent drifts on multi-step work, invest in tools that expose current state, next actions, and invariants in structured output. A thin CLI or stronger type surface can buy more than another prompt paragraph.

      Attribution:
    • NitpickLawyer #1
    • nicoty #1
    • Vgoose #1
  3. 03

    Separate refactor passes beat one-shot cleanup prompts

    Trying to get a model to implement features and continuously simplify the code at the same time is producing weak results. The stronger pattern is to split those jobs. People reported better outcomes from periodic whole-codebase review, dedicated refactor turns, and adversarial second passes that hunt for rough edges after the feature lands. That matches the benchmark’s finding that a simple 'plan first' instruction does little on its own.

    Design your agent workflow as a loop with explicit cleanup phases, not a single implementation pass plus a polite request to refactor. Measure the effect of each extra review stage with your own evals so you know when it is reducing complexity versus just adding churn.

      Attribution:
    • dmrivers #1
    • jrflo #1
    • piazz #1
    • anentropic #1
    • thatfunkymunki #1
  4. 04

    Harness design can create slop by itself

    Some of the bloat blamed on Opus 5 may be self-inflicted by the orchestration layer. One concrete report described review subagents that became so pedantic they turned a one-table SQL migration into a 200-line script through feedback loops. Another commenter cut closer to the root. If the agent can touch anything, slop accumulates. Constraining edits to one seam and preferring additive changes over broad in-place rewrites can matter more than model selection.

    Audit your harness before declaring a model regression. Put hard bounds on where an agent can edit, and watch for review loops that reward performative complexity instead of better code.

      Attribution:
    • patwolf #1
    • latentsea #1
    • delbertty #1
  5. 05

    Code cleanliness now affects AI throughput too

    The maintainability argument is no longer just about pleasing future human teammates. Commenters pointed to recent papers claiming that cleaner code also improves coding-agent efficiency and navigation. That makes tech debt a direct tax on your AI tooling, not just on staff velocity. Slop matters because future modifications get harder, and the benchmark is explicitly trying to measure that effect rather than treating ugliness as a matter of taste.

    Treat maintainability work as infrastructure for both humans and agents. If you plan to rely on AI in an existing codebase, expect dirty code to raise cost and reduce reliability even when current defects look acceptable.

      Attribution:
    • jdm2212 #1
    • paffdragon #1
    • Majromax #1
    • adamtaylor_13 #1

Against the grain

  1. 01

    The posted result may not be statistically meaningful

    The variance reporting in the original paper was loose enough that the claimed differences in cost and quality may not clear a real significance bar. If the plus-minus values are standard deviations, the overlapping error bars could mean the models are effectively tied on those measures. That does not invalidate the benchmark, but it does weaken any strong ranking claims drawn from the numbers shown.

    Use this benchmark directionally, not as a precise leaderboard, until variance and confidence are reported more cleanly. If you are making model-buying or training decisions, rerun enough samples to estimate your own uncertainty.

      Attribution:
    • jrflo #1
  2. 02

    More functions is not automatically worse code

    Function count and line count can mislead if they are treated as proxies for maintainability on their own. Breaking logic into smaller callables can improve readability and testability, and raw line reduction can even push code toward dense one-liners. Several comments pushed for structure-aware measures such as Abstract Syntax Tree based complexity instead of assuming 'more functions' equals 'more slop.'

    Do not turn one surface metric into a policy target for your agents. Track complexity with multiple signals and inspect whether decomposition is clarifying the design or just fragmenting it.

      Attribution:
    • falsemyrmidon #1
    • Normal_gaussian #1
    • drcongo #1
  3. 03

    Perceived model degradation may be user adaptation

    Claims that labs quietly worsen models after launch got a lot of airtime, but the strongest pushback was that users are terrible model evaluators. As models improve, people raise task difficulty, shorten prompts, and normalize the new baseline, so later failures feel like regression. Historical tracker data was cited as evidence that releases generally do improve over time, even if anecdotes on subscription products keep feeding the downgrade story.

    Do not rely on vibes when deciding that a model got worse. Keep stable internal evals across model versions and separate API results from subscription product behavior before changing vendors or workflows.

      Attribution:
    • ACCount37 #1 #2
    • xnorswap #1
    • svnt #1
    • jkman #1

In plain english

CLI ↩
Command-line interface, a text-based tool or program that you run from a terminal.
SlopCodeBench ↩
A benchmark for coding agents that tests whether they can keep a codebase workable across a series of feature additions, not just solve one isolated task.

Reference links

Benchmarks and benchmark-related references

  • Do skills improve coding agent accuracy
    Referenced as another evaluation that uses SlopCodeBench to assess whether skill packs help or hurt agent performance.
  • SlopCodeBench paper
    The benchmark paper used to set up the evaluation in the post and discussed throughout the comments.
  • Cognition FrontierCode
    Mentioned as a benchmark suggesting medium effort may be a better quality-cost tradeoff, and contrasted with SlopCodeBench's design.

State, planning, and agent tooling

  • agent-scaffold
    A project for structured plan templates, validation, and state-machine-like workflow support for agents.
  • OpenSpec
    Suggested as an existing tool that already implements structured planning and state handling ideas discussed in the comments.
  • Types with AI
    Shared as a writeup on using type information to improve communication and state visibility for AI systems.

Model performance tracking and model behavior

Papers and concepts on maintainability

Analogy and skepticism references