A farmer’s question tests more than an agricultural language model
Recent CGIAR and IFPRI work shows why an advisory evaluation should follow the question through language, references, interface and human understanding.
A useful agricultural assistant must understand what is being asked before it can offer a useful answer. A farmer asking whether a notice applies to this season may need its date and district, rather than another general explanation of the crop. An extension worker may need to compare two conflicting passages. Treating both requests as ordinary question answering can produce a fluent response while missing the practical task.
That distinction runs through IFPRI’s May 2026 account of agricultural AI benchmarking. It calls for evaluating models, systems and processes, including usefulness, usability, language and equity. The account discusses a November 2025 community webinar; it is an evaluation argument, not a new field trial proving better harvests. Its value is the larger unit of inspection: the entire advisory encounter. [IFPRI’s evaluation account](https://www.cgiar.org/news-events/news/beyond-model-evaluating-ai-agricultural-advisory-systems-so-they-work-field).
Development progress deserves an accurate label
CGIAR’s July 2025 AgriLLM article describes workshops at ILRI in Nairobi that developed questions for different user roles and assembled evidence-based answers. This was foundational development work. It shows attention to agricultural context and reference material, but workshop-generated answers are not observations of farmers using a completed service. [CGIAR’s development account](https://www.cgiar.org/news-events/news/agrillm-how-cgiar-is-developing-an-ai-powered-agricultural-advisory-service-for-global-south).
The distinction remains useful when reading newer portfolio information. CGIAR’s living AI Hub page describes AgriLLM as an MVP launched in December 2025, with further evaluation work to follow. The milestone belongs to the product’s history; it is not the page’s publication date or evidence that every language and crop has been evaluated. [CGIAR’s stated product status](https://www.cgiar.org/cgiar-research-portfolio-2025-2030/digital-transformation/cgiar-ai-hub).
Digital Green’s FarmerChat page advertises voice, text and photo inputs. Those modes illustrate why a text-only benchmark is insufficient: the service has to receive a question through an interface before a language model can answer it. This is a first-party feature description, not an independent endorsement, and its promotional cost and universal-language statements are not adopted here. [FarmerChat’s described input modes](https://www.digitalgreen.org/farmerchat).
Make the question inspectable
Consider a fictional adult extension group near Kisumu preparing a demonstration with synthetic records. Nobody submits a real farmer’s identity, precise plot location or private voice recording. The group creates a small reference pack containing an invented crop calendar, a dated meeting notice and a deliberately outdated advisory. The exercise asks whether an assistant can explain which document applies and identify the missing information.
Before trying any system, the group writes down what a satisfactory response would contain. For the meeting notice, that means the correct date, location and source. For the conflicting advisories, it means acknowledging the disagreement rather than silently choosing the more recent-looking paragraph. For a question outside the reference pack, it means explaining the gap and identifying an appropriate human contact. No chemical rate or treatment decision is part of this exercise.
A question should travel with its intended meaning. An evaluator records the farmer’s wording, the relevant crop stage if supplied, and the detail needed to distinguish similar questions. The reference answer explains which information is essential and which is optional. Otherwise a terse but correct clarification can be marked worse than a long, unsupported answer because the scoring sheet rewards length or confidence.
Separate the listening problem from the answering problem
For a voice interface, use a fictional sentence whose written meaning is already agreed. Ask a consenting adult colleague to read it, then compare the transcription with that meaning. If the system hears a different crop or date, record that as an input failure. Do not hide it inside a single overall answer score. Conversely, a correct transcription does not excuse an answer that invents a source.
Language review needs a reader who can judge meaning in context. Translating a response back into English can be a diagnostic step, but the original user’s understanding matters. The group can ask an adult participant to explain the answer in their own words and point to its conditions. A response that sounds natural yet drops a warning or time limit should be revised before anyone treats it as advice.
A practical evaluation record therefore keeps several observations beside each question: what entered the system, what the interface displayed or played, which reference supported the answer, and what the participant understood. These observations can disagree. Keeping them separate makes improvement more precise: repair the microphone flow, revise the reference collection, or change the response wording, rather than replacing a model without knowing the fault.
Build difficult cases into the exercise
The fictional pack should include a question with an unknown village, a date that has already passed, an unsupported crop, and two documents that use the same term differently. Include an ordinary misspelling and a request that needs clarification. These are proposed evaluation cases, not a published benchmark or a claim about a named service’s failures. Their purpose is to expose how the service behaves when the answer is not conveniently available.
Keep some cases out of routine demonstration and prompt editing. If a team continually rewrites its reference answers after seeing the system’s output, it loses a useful check on whether changes generalize. Record which cases influenced the change, which remained separate, and the system version used. A later model or retrieval update should not inherit the earlier result automatically.
The comparison should also include the existing way of finding information. In this exercise, an adult might locate the notice directly from the reference pack with help from an extension colleague. Observe the steps and any misunderstandings for both approaches. The aim is to learn whether an assistant improves the defined task, not to prove that a chatbot is preferable merely because it exists.
Let uncertainty lead somewhere useful
An unavailable service should preserve a safe draft question where appropriate and explain how to try again. An unsupported answer should identify the missing fact and a realistic next step. The group should inspect that handoff too: can the adult find the cited document or reach the responsible adviser? A polished refusal without a usable path can still leave the original task unresolved.
The resulting record is modest but valuable. It describes the questions, languages, reference editions, interface conditions and human observations actually examined. A larger field evaluation would need a separately designed method for adoption and agricultural outcomes. Neither a successful demonstration nor enthusiastic feedback supplies that evidence by itself. This article offers educational evaluation ideas, not agronomic advice, a recommendation to buy a named product or a report of a real deployment.
Evidence diagram and readable guide
Original explanatory diagram: a fictional adult advisory exercise separates question meaning, reference support and comparison; no service or agricultural outcome is endorsed.
[Open the complete evidence diagram](/assets/agriculture-workflow-advisory-evaluation-20261015.svg)
Read the diagram
**A farmer’s question tests more than an agricultural language model**
A proposed evaluation with fictional adults and synthetic records; no named service was tested or endorsed.
- **Defined farmer question** Record the practical task and missing context.
- **Input meaning** Inspect transcription and language meaning.
- **Trusted reference pack** Preserve the relevant source and edition.
- **Proposed answer** Keep supported statements and gaps visible.
- **Adult comparison** Check understanding and source access.
- **Scoped evaluation record** Retain cases, versions and unanswered questions.
Input interpretation and reference support are separate inputs. A correct transcription does not prove a supported answer.
How the connections work
- The defined question establishes what the input interpretation must preserve.
- The interpreted input informs a proposed answer without replacing its source.
- The reference pack separately constrains what the proposed answer can support.
- An adult compares the proposed answer with its source and intended meaning.
- The comparison is recorded within its cases and conditions, not as a harvest result.
Relationship key
Two inputs meet at the proposed answer: interpreted meaning and a trusted reference. Adult comparison leads to a scoped record, without implying a harvest benefit.