A multilingual AI score needs a language, a task and a release record
Sarvam’s March 2026 weight release and August API retirement frame a practical question: what language task was tested, with which artifact and evaluator?
Sarvam's 2026 model story contains two developments that should stay on separate lines. Its [6 March announcement](https://www.sarvam.ai/blogs/sarvam-30b-105b) introduced downloadable Sarvam 30B and 105B models. Its [18 August API changelog](https://docs.sarvam.ai/changelog/2026/8/18) later said sarvam-30b was no longer available through the hosted Chat Completions API. A retained weight artifact and an operating hosted endpoint are different forms of access.
For a multilingual project, neither form answers the next question: does this exact system handle the language and task people actually need? That requires an evaluation record. A familiar model name, a fluent greeting or an aggregate leaderboard position cannot stand in for it.
Model rights follow the release
The official [sarvam-30b model card](https://huggingface.co/sarvamai/sarvam-30b) identifies Apache 2.0. The separate [sarvam-1 license](https://huggingface.co/sarvamai/sarvam-1/blob/main/LICENSE.md) requires explicit written permission for commercial use. These are different artifacts; a provider's name does not give all of its releases one license.
The earlier [sarvam-1 card](https://huggingface.co/sarvamai/sarvam-1) describes an Indian-language focus. That is a reason to examine its stated scope, not a guarantee for every Indic language, regional expression or workflow. A team should establish whether its intended experiment is permitted before downloading or evaluating a candidate.
What an institutional benchmark contributes
[IndicGenBench](https://arxiv.org/abs/2404.16816), introduced in a 2024 paper, covers generation tasks across 29 Indic languages, 13 scripts and four language families. Its tasks include translation, summarisation and question answering. This provides a more specific starting point than asking whether a model is multilingual in the abstract.
The [authors' dataset repository](https://github.com/google-research-datasets/indic-gen-bench) also separates component licenses: CrossSum-IN, for example, has non-commercial share-alike terms, while other components use different terms. Reusing a benchmark therefore needs its own provenance record. The model's license is not the benchmark dataset's license.
A fictional college chooses a narrower question
Consider an adult technical college in Bengaluru preparing a research demonstration about support-ticket summaries in Hindi and Tamil. This is a fictional exercise, not an account of a Sarvam deployment. The college uses invented tickets, written and checked by competent speakers, and makes no claim about workplace performance or student attainment.
Each ticket contains an equipment observation, a requested next step and a detail that must remain uncertain. The demonstration asks whether a model can summarise those three elements without turning a reported symptom into a confirmed cause. It does not ask the model to approve maintenance or decide a student's access to a service.
Separate script from comprehension
The teachers create distinct groups: one language in its usual script, informal mixed-language text, and a short note containing a spelling variation. These groups are pedagogical test categories, not a claim that the published benchmark covers each one. A model that produces the expected script may still have misunderstood who observed the fault.
Reviewers examine whether the summary preserves the person, event and uncertainty in the source. They also look for omissions that change the practical meaning. An apparently polished summary can be less useful than a plain one if it quietly removes the request for a callback or changes a tentative date into a deadline.
Keep a language-specific result
A combined pass rate could conceal that the system handles one group well and another poorly. The college therefore keeps results by language, script condition and task. It records how many examples each result represents. A handful of classroom examples cannot support a national or worldwide performance claim.
Reviewers should be allowed to disagree and explain why. Some differences concern style, while others concern meaning or an unsupported assertion. Treating all disagreements as a single error count loses the very information that would help the team improve the exercise or decide that the task should remain manual.
The evaluator is part of the experiment
The [Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) provides configurable evaluation machinery. Its official README records a September 2026 development allowing plugins for backends, filters, metrics and aggregations. That is a concrete change in how an evaluation can be extended, not evidence that a new plugin's scores are automatically correct.
For the college, a custom metric is itself reviewable code. The team retains its version and checks a few expected cases by hand. If an answer-extraction rule throws away uncertainty or awards credit for a keyword in the wrong sentence, the resulting score can reward the wrong behaviour.
Preserve the full comparison conditions
The run record identifies the model revision, tokenizer, prompt template, generation settings and evaluation configuration. It states whether a hosted service or a local weight artifact produced the answer. A retired endpoint cannot be treated as a reproducible implementation simply because an old notebook still names it.
The team also records failures to load, incomplete outputs and interrupted requests. It does not remove those cases from the denominator without explaining the decision. An evaluation of only the answers that successfully appeared on screen is different from an evaluation of all requested work.
Human review needs material it can judge
The fictional ticket set includes an answer guide, the supplied source and a reason for each expected element. Reviewers receive enough context to distinguish faithful compression from an unsupported inference. Asking them only whether a response sounds good would measure a different property.
If two formulations are equally faithful, the guide should allow both. If the source itself is ambiguous, that should be recorded rather than repaired by inventing a fact. The purpose is to learn what the system does with evidence, not to force every answer into one preferred sentence.
Extend worldwide by adding evidence
A college in Recife could adapt the structure for Portuguese, while an organisation in Nairobi could design a separate Swahili exercise with qualified reviewers. Those are proposed adaptations, not claims that the named models support either use successfully. Local terminology and the available evaluation material would need fresh examination.
The same discipline applies within India. A result in one language does not become a result in another because both appear in a model card. A published benchmark and a locally reviewed task set can complement each other, but their results should not be casually merged into a single universal quality number.
Access and quality deserve separate decisions
The March release and August endpoint retirement show why an evaluation record needs both an artifact identity and a delivery route. A team can preserve a reproducible research candidate while finding that its former hosted path is unavailable. It can also find that an available model does not meet a particular language task.
This is educational, non-promotional reporting checked on 15 October 2026. Provider release and retirement statements are attributed to their dated records; no model was run and no benchmark score was reproduced here. Commercial rights, actual endpoint readiness and language-task acceptance remain separate questions.
Evidence diagram and readable guide
Original explanatory diagram: dated access records and evaluation inputs remain separate from language-task results.
[Open the complete evidence diagram](/assets/open-source-workflow-language-evaluation-20261015.svg)
Read the diagram
**Separate access from language quality**
Release and service-retirement records are parallel access facts, not causes of language quality. All classroom cases are fictional.
- **March 6: weight announcement.** Sarvam’s dated 2026 release account introduces downloads.
- **August 18: hosted30B retirement.** The dated 2026 changelog concerns its hosted Chat Completions route.
- **Permitted language-task material.** Inspect each dataset component and intended use.
- **Named artifact and evaluator.** Retain model revision, tokenizer, prompt and metric code.
- **Reviewed comparison cases.** Use qualified reviewers and preserve ambiguity.
- **Results by language and task.** Do not pool distinct languages or missing outputs into a universal score.
Dashed outlines group parallel inputs. Arrows carry information for review, not a measured effect.