A legal-AI benchmark measures a task, not a licence to handle a matter
LegalBench and SG-LegalCite distinguish reasoning tasks, retrieval targets and the untested work around a complete legal matter.
A system can answer a legal multiple-choice question, find a related judgment and draft a fluent explanation while performing differently at each step. Treating those achievements as a single quantity called legal intelligence makes evaluation easier to advertise and harder to interpret.
LegalBench and the newer SG-LegalCite research offer concrete alternatives. One organises different kinds of legal reasoning tasks; the other asks how an explicit legal principle changes citation retrieval in Singapore law. Their value is not a universal league table. It is a more precise account of the work being tested and the evidence a score leaves outside the frame.
LegalBench separates kinds of reasoning
The [LegalBench paper first submitted on 20 August 2023](https://arxiv.org/abs/2308.11462) describes 162 tasks covering six types of legal reasoning and an evaluation of 20 open-source and commercial language models. Legal professionals helped construct the tasks. This is a dated research record, not a current test of every model available in October 2026.
The same programme appears in the [NeurIPS 2023 Datasets and Benchmarks proceedings](https://proceedings.nips.cc/paper_files/paper/2023/hash/89e44582fd28ddfea1ea4dcb0ebbf4b0-Abstract-Datasets_and_Benchmarks.html). That publication is not a second study. The figures count different things: tasks, reasoning categories and evaluated models. None is a count of clients, completed legal matters or practising lawyers replaced. A model's performance on these tasks cannot by itself establish that it can manage an unfamiliar dispute, communicate appropriately with a person or recognise every missing fact.
The task page explains what a score refers to
The project's [task descriptions](https://hazyresearch.stanford.edu/legalbench/tasks/) distinguish categories including issue spotting, rule recall, application, conclusion, interpretation and rhetorical understanding. Individual tasks use defined input-output arrangements. The page is part of the same LegalBench programme, not an independent replication of its results.
That structure helps legal professionals ask better evaluation questions. Recognising whether a passage raises an issue differs from finding the applicable authority or explaining its consequences in a new situation. A broad average can obscure those distinctions, especially when the intended work resembles only a small part of the benchmark.
Singapore retrieval research puts the principle into the query
The [SG-LegalCite preprint submitted on 20 May 2026](https://arxiv.org/abs/2605.21057) describes 100,890 case-principle pairs drawn from 8,523 Singapore Supreme Court judgments spanning 2000 to 2025. It evaluates 11 baselines and investigates retrieval using facts together with explicit legal principles.
The paper's premise is that factual similarity alone can retrieve material that misses the doctrinal issue. Its dataset counts pairs rather than independent cases or successful legal outcomes. This article reports the benchmark design and scale from the public research record; it does not reproduce the experiments or claim that any tested system is fit to provide legal advice.
Compare units before comparing results
| Record or activity | Unit being examined | Question still outside the result | | --- | --- | --- | | LegalBench | Defined legal-reasoning tasks | Performance across an entire real matter | | LegalBench task catalogue | Inputs, outputs and categories | Independent confirmation of scores | | SG-LegalCite | Retrieval of case-principle relationships | Current authority treatment and legal suitability | | Fictional workflow review | A sequence of proposed tasks | A verified service-level success rate |
This is an original comparison table. It does not rank products or pool scores across incompatible tests. Different output formats, source collections and scoring rules can answer different questions even when each study concerns legal language.
A fictional developer-lawyer workshop
Imagine a fictional adult workshop in Singapore involving legal researchers and software developers. They use a small invented rulebook about allocating rooms in a training centre. The rules, exceptions and sample decisions are written for the exercise and have no legal force. No real case or client material is processed.
The group creates three tasks. The first asks which rule is relevant. The second finds a sample decision addressing the same exception. The third explains how the exception changes a proposed allocation. A system that succeeds at one task may fail at another, and the team can inspect why without advising anyone about a real matter.
Write the intended proposition before searching
In the fictional record, two decisions share the same room and date but concern different exceptions. Another uses different facts while addressing the principle the trainee needs. The group compares a search based on surface similarity with one that states the intended principle explicitly.
This is an illustration of a research question, not a replication of SG-LegalCite or a measured improvement. It makes the target inspectable: the system is being asked to retrieve support for a particular proposition. A related-looking document is not automatically a successful answer, even when it is authentic and easy to read.
The researchers keep a reference explanation of why each sample decision is relevant or irrelevant. Disagreement over that reference is recorded rather than hidden in a single score. This reveals that defining a task can require substantial domain judgment before any model is run.
Abstention needs a place in the test
The workshop includes a question the invented rulebook does not answer. The team examines whether the assistant recognises the gap or manufactures a confident rule. It also includes a request with two plausible interpretations, where clarification would be more useful than an immediate conclusion.
These cases are proposed evaluation additions, not claims about the coverage of either published benchmark. They help distinguish output generation from appropriate behaviour in an information gap. A service-level assessment might need many more conditions, including changes in sources and the way a person responds to a request for clarification.
The full workflow introduces new failure points
The fictional team joins retrieval to drafting. A relevant passage can still be paraphrased incorrectly; an accurate explanation can still omit a qualification; a correct draft can still be associated with the wrong record. Measuring the retrieval step alone would leave those later events untested.
The interface therefore keeps the retrieved passage, the proposed explanation and the trainee's accepted revision separate. If processing stops, it preserves the work and explains what remains incomplete. These are original design proposals. No application was built, tested or accepted as part of the source research described in this article.
A task score is not a staffing calculation
Benchmarks can help identify capabilities worth investigating. They do not measure how much professional work remains in defining the question, finding missing information, maintaining the source collection or reviewing an explanation. Nor do the reported dataset sizes tell us how many people a service will help.
For legal teams, useful task redesign may involve moving effort from searching towards specifying and checking the proposition. For developers, it may involve exposing the evidence path and supporting uncertainty. Those are possible directions for evaluation, not verified labour-market outcomes or a prediction that a fixed proportion of jobs will disappear.
What a more informative claim sounds like
A defensible description names the task, source population, model version, test conditions and scoring method. It keeps historical experiments separate from current service performance and describes what was not tested. That makes the claim easier for a legal professional or a prospective user to evaluate.
LegalBench and SG-LegalCite provide different research tools for that conversation. Neither grants professional authority or settles suitability for every jurisdiction. Their useful contribution is specificity: reasoning categories, a retrieval target and observable limits. This is general educational reporting checked on 15 October 2026, not individual legal advice, a model recommendation or a claim to professional credentials.
Evidence diagram and readable guide
Separate LegalBench and SG-LegalCite unit panels precede an illustrative evaluation workflow, without a shared performance ranking.
[Open the complete evidence diagram](/assets/legal-workflow-benchmark-work-20261015.svg)
Read the diagram
**Name the task before reading the score**
Different benchmark units answer different questions. Neither grants professional authority.
- **LegalBench, 2023.** 162 tasks. Six reasoning categories. 20 evaluated models. Paper, proceedings and task catalogue: one programme.
- **SG-LegalCite v1.** 20 May 2026 preprint. 100,890 case-principle pairs from 8,523 judgments, 2000–2025. 11 baselines.
Related-looking facts need not support the same legal proposition. The retrieval research examines facts plus an explicit principle.
Illustrative evaluation workflow
- **Define the task.** State the input, expected output, model version and test conditions.
- **State the intended proposition.** Write what the retrieved material is meant to support.
- **Retrieve evidence.** Inspect relevance rather than surface similarity alone.
- **Inspect the explanation.** An authentic source can still be paraphrased incorrectly or lose a qualification.
- **Test information gaps.** Include ambiguity and unanswered questions. Clarification or abstention needs a place in the review.
These are proposed workshop steps, not a reproduction of either benchmark or a measured service-level success rate.
**Reading the evidence.** Tasks, pairs, judgments and successful legal matters are different units. Historical research counts do not establish a current product ranking, service suitability or staffing saving.
*Sources: LegalBench paper, NeurIPS 2023 and task catalogue; SG-LegalCite v1, May 2026.*
**Keep the comparisons separate.** Different tasks, corpora and denominators; no common performance axis.