Harness engineering for self-improvement

The strongest reaction was not philosophical. It was operational. People already doing this said the hard part is not getting an agent to tweak its setup. It is defining a fitness function that matches useful work in your own repo. Several comments converged on the same pattern. Start with private, repo-specific evals instead of public benchmarks. Use tests because they are the best deterministic signal available, but do not confuse test passing with code quality or maintainability. Getting runnable environments for arbitrary repos is still painful, and selecting tasks that actually discriminate between agents is harder than it sounds. A repeated warning was that weak evaluators get gamed fast. Incomplete check suites and scalar rewards invite cheating, benchmark overfitting, and fake progress. A second theme was that many current harnesses are overbuilt. Multiple people reported better results after shrinking the context and tool surface rather than adding more instructions, more MCP servers, and fatter system prompts. One concrete report claimed the same hidden-graded task passed with fewer tokens, fewer calls, lower memory use, and faster completion when the setup was reduced to a shell tool and less prompt scaffolding. Others refined that into a more useful rule. Keep always-on context small, and move infrequent policies into separate docs that the agent can pull in when needed. The point was not that standards and memories do not matter. It was that indiscriminate context stuffing often makes agents worse. The most practical comments treated harness work like ordinary software engineering. People run agent retros after sessions to capture pain points, missing tools, unclear docs, and possible invariants, then feed that back into scripts, lint rules, skills, or repo guidance. Others mine production traces, let agents build helper tools, and use validation splits so the harness does not simply learn the test. Cost and quality wins came from unglamorous changes like quiet terminal output, codebase indexing, and narrowing the optimization target to a small harness component instead of trying to optimize a giant codebase end to end. The thread landed on a blunt view of where the field is. Harnesses clearly matter and can produce real gains right now. But the gains come from disciplined measurement, narrower scope, and ruthless simplification, not from hand-wavy claims of recursive self-improvement.