The post introduces Terminal-Bench-Science, a benchmark for AI agents doing real scientific work in a terminal. Instead of short benchmark questions, it gives models markdown instructions, data, and libraries inside a Docker container, then scores the result with deterministic pytest-based verifiers that allow for numerical tolerances and multiple valid implementations. That design landed well. People have wanted something closer to actual research workflows than the usual toy agent tasks.
The main reaction was approval of the direction, with a hard caveat about trust. Several readers said the task list is the real artifact here, not the leaderboard, because the value of a benchmark like this lives or dies on whether the tasks actually resemble research work and whether the checkers catch wrong-but-plausible output. The project maintainers pushed back on the idea that correctness was hand-waved, saying the benchmark is built around outcome verification rather than matching one reference solution. Even with that, the mood was that passing a benchmark is still far from being safe to trust in a paper or lab workflow. The gap people care about is not just raw problem solving. It is whether the model follows the exact method requested, stays on track over long horizons, and avoids quietly swapping in simpler approaches.
The model ranking itself drew less confidence than the benchmark design. Some readers were surprised that Opus beat Fable on scientific tasks despite opposite coding impressions. Others pointed out that personal anecdotes about which model has more "scientific intelligence" are mostly vibes until grounded in public tasks and reproducible evals. A few commenters also noted a likely downside of making all tasks public. Once model makers train directly against them, scores may jump and the benchmark may lose value as a measuring stick. So the takeaway was not "model X won." It was that domain-specific, workflow-level evals are finally getting serious, and now the hard part is keeping them representative, trustworthy, and hard to overfit.
If you care about AI for R&D, pay more attention to workflow-level evals like this than generic coding leaderboards. But do not treat headline scores as proof a model is ready for paper-grade work until you inspect the task set, failure modes, and how easily the benchmark can be gamed.
Mostly positive about the benchmark’s direction and overdue focus on real research workflows, but skeptical about reliability claims and wary that public tasks will quickly get gamed. People liked the move away from toy benches more than they trusted any single leaderboard result.
Key insights
01
Pytest verifiers are doing the heavy lifting
The benchmark is not grading style or matching one gold script. It runs deterministic pytest checks on the final outcome inside a Docker sandbox, with tolerances for simulation noise and room for multiple valid implementations. That is a much stronger setup than many agent benches, but it still leaves open the failure people actually fear in science. A model can pass the verifier and still have ignored the requested method or produced work you would not trust in a paper.
When you evaluate agent systems for R&D, inspect the verifier design before you look at the scores. Add explicit checks for method compliance and instruction following if the exact procedure matters, not just the final numeric output.
The strongest endorsement was for the task library itself. All tasks live in a public GitHub repo, were contributed through pull requests, and can be extended into new domains like RF engineering. That makes this less useful as a permanent leaderboard and more useful as shared infrastructure for building better domain evals. The benchmark will only stay relevant if practitioners keep adding hard, field-specific tasks that break current models.
If your company works in a specialized domain, do not wait for generic benchmarks to represent you. Contribute or clone the task framework and build your own eval set around the workflows that actually block your team.
Scientific agents need stability more than flashes of brilliance
The comments that best explained the surprising model rankings argued that long-horizon scientific work rewards steadiness, context control, and low-confabulation behavior more than raw cleverness. One reader described Fable as strong on deep debugging but unreliable when given freedom to roam, while Opus looked less creative but more governable. Another pointed out that biomedical guardrails can also tank practical usefulness on science tasks. That framing makes the leaderboard easier to read. Breadth, persistence, and refusal behavior can beat a model that looks smarter in short coding sessions.
For research automation, benchmark and buy for long-horizon control, not just peak reasoning demos. Run your own tests that expose drift, refusals, and context loss over multi-step workflows.
Several people challenged confident claims that one model clearly has better scientific or mathematical instincts than another. One reader reported Sol saturating a benchmark on flawed mathematical proofs where Fable scored below 50 percent, which directly cut against earlier impressions. The useful point is not which anecdote wins. It is that domain performance is sharply task-dependent, and generic impressions from coding or chat use do not transfer cleanly to research work.
Do not standardize on a model because it feels smarter in everyday use. Build a small internal benchmark from your own artifacts and compare models on that before making workflow or procurement decisions.
Open benchmarks decay fast once labs train on them
A blunt criticism was that public release may shorten the benchmark’s shelf life. If model providers can see every task and verifier, the next round of models may post big gains that reflect targeted training rather than broader scientific competence. That does not kill the project, but it changes what the scores mean. You should treat future improvements as partly a measure of benchmark exposure, not just underlying capability.
Use public benchmarks for transparency and ecosystem building, but keep a private holdout set for any decision that matters. Without hidden tasks, you cannot tell generalization from benchmark tuning.
Better science throughput could mean more science slop
The strongest pushback was not about benchmark quality. It was about the target itself. Automating scientific workflows may flood literature and review pipelines with plausible but weak work, which is a worse failure mode than bad code because frontier research has fewer established patterns and less robust verification. That reframes the whole effort. A benchmark that makes models better at producing research artifacts may create institutional noise before it creates reliable discovery.
If you deploy AI into research production, invest in review bottlenecks and provenance tooling now. More output is not a win if your team cannot cheaply separate solid work from polished nonsense.
Software systems built on AI models that can take actions over multiple steps, such as reading files, running commands, and editing code to complete a task.
The code or logic used to judge whether a model’s output successfully completed a task.
Reference links
Benchmark and task repository
Terminal-Bench-Science task directory
Pointed to as the part of the project readers should inspect first because the task set determines whether the benchmark is meaningful.
terminal-bench-science.ai