Stanford Law’s press release sold this as “AI outperforms law professors,” but the paper itself tested something much narrower. Sixteen law professors wrote short answers to common first-year contracts questions, then professors blind-compared those answers against outputs from Gemini 2.5 Pro and NotebookLM and chose which one they would rather give to a student. On that preference measure, the AI answers won roughly three quarters of the time and were flagged as less pedagogically harmful. That is a tutoring and presentation result, not proof that an LLM can practice law better than a lawyer.
Most of the useful discussion landed on methodology. The setup appears optimized for what LLMs already do well: produce polished, confident, well-structured prose in response to textbook-style prompts. Professors were told to be concise and spent only a few minutes per answer, which likely compressed the humans toward terse drafts while the models stayed fluent and complete. Several readers pointed to the paper’s own appendix showing answer length as the strongest predictor of winning. Others argued the headline overreaches twice, first by collapsing “law professors preferred these tutoring answers” into “AI outperformed law professors,” and second by inviting readers to generalize from
1L contracts teaching to legal practice.
The sharper comments from lawyers and people doing real legal work were less interested in the press-release framing and more interested in where LLMs actually help. The consensus was that they are already useful for legal research, brainstorming arguments, first-pass drafting, and surfacing niche cases. The failure mode is not just fake citations. It is more dangerous when the citation is real but irrelevant, outdated, or misread for the jurisdiction and procedural posture at hand. That is why legal work is harder to automate safely than coding. Code has tests, runtimes, logs, and faster feedback loops. Legal documents often reveal errors only after filing, signing, or litigation, when fixes are expensive or impossible.
That led to the practical conclusion: LLMs look more like aggressive paralegals or tutors than autonomous lawyers. They can save time for experts who know the domain well enough to constrain and verify them. They are risky for juniors and non-experts because confident output hides mistakes, and repeated success makes reviewers lazy. A smaller but persistent current in the comments said the broader direction still looks obvious. Even if this paper is sloppy, law is full of text retrieval, synthesis, and drafting, which matches the strengths of current models. The moat shifts from raw text production to trusted workflows built around retrieval, validation, local court norms, and human accountability.
law.stanford.edu