New study tests how consistently AI models grade medical students’ clinical summaries

A published comparison examines repeated ratings, human agreement and prompt design using virtual-patient case summaries.

A study published in JMIR Medical Education on 21 October compares how language models score medical students’ written clinical summaries, including whether their ratings remain stable when repeated. The research used 122 summary statements, six rubric components and multiple model-and-prompt conditions.

The experiments were conducted in 2025, so the paper is a new publication about that model generation rather than a ranking of today’s newest systems. Its central finding is that consistency and agreement with human experts depend on the model, prompt and particular part of the rubric.

The authors have made code, prompts, cases and reference ratings available through a public repository, providing a concrete basis for further assessment work.

A concise summary contains several different skills

A clinical summary is more than a shortened case description. The learner has to select important information, transform it into useful clinical language and narrow the problem without changing the facts.

A rubric separates those tasks so that feedback can be specific. Someone may preserve factual details while failing to prioritise them, or produce a concise account that omits an important distinction. A single overall score can conceal that difference.

For an educational system, component-level scoring therefore offers a useful design. The learner can see which aspect needs work and revise the summary accordingly. The teacher can also examine whether the feedback follows the intended learning objective.

Repetition reveals another part of performance

A model can produce a plausible grade on one attempt yet behave differently when asked again. Testing repeated ratings makes that variation visible.

For a classroom product, the practical issue is the experience of the learner. Two similar submissions should receive feedback that reflects their content, rather than changes caused by a hidden prompt variation or a different run.

That suggests a useful development sequence: write the rubric clearly, prepare representative examples, compare model ratings with expert judgments and repeat the exercise. Disagreements can then be examined at the level of an actual statement and scoring rule.

The study also highlights the difference between format compliance and assessment quality. A response can contain a valid number while applying the wrong standard. Structured output helps software process the result, but the underlying judgment still needs evaluation.

Reusable materials support better comparisons

The availability of prompts and reference ratings gives other teams a way to reproduce the task instead of relying on a short description of it. A new model can be assessed against the same materials, while a revised rubric can be tested as an explicit change.

That is particularly useful when educational tools are updated regularly. The team can see whether a change improved feedback on the intended skill and whether earlier strengths were retained.

The publication’s contribution is therefore a well-defined assessment problem: a case, a learner’s summary, a rubric and a repeatable comparison with human scoring. It gives educators and developers a more concrete foundation for AI-assisted feedback than a general claim that a model can grade student work.

The next application decision is how to place that feedback within teaching, so students can use it to revise their reasoning and faculty can understand how the scoring behaves.

Sources

- [Test-Retest Reliability of Large Language Models and Their Concordance With Human Experts in Rubric-Based Scoring of Clinical Summary Statements](https://mededu.jmir.org/2026/1/e94666) — JMIR Medical Education · 2026-10-21 - [MedicalSummaryStatements_LLMs](https://github.com/DSFZ-Team/MedicalSummaryStatements_LLMs) — DSFZ Team · accessed 2026-10-26