Singapore’s AI testing toolkit is a starting point for a real test plan

Project Moonshot supplies actual open-source evaluation code. The application owner still chooses relevant cases and reviews the results.

A team evaluating an AI application needs more than a list of abstract risks. It needs a way to run cases repeatedly, record outputs and compare configurations. Singapore’s Project Moonshot provides an actual open-source toolkit for that work. Its value depends on the test plan and application around it, rather than on the mere presence of a recognised tool name.

IMDA’s [May 2024 launch announcement](https://www.imda.gov.sg/resources/press-releases-factsheets-and-speeches/press-releases/2024/sg-launches-project-moonshot) introduced Moonshot. Its [January 2026 starter kit](https://www.imda.gov.sg/-/media/imda/files/about/emerging-tech-and-research/artificial-intelligence/starter-kit-for-testing-llm-based-applications-for-safety-and-reliability.pdf) addresses testing LLM applications. The [AI Verify Foundation repository](https://github.com/aiverify-foundation/moonshot) contains the implementation and identifies an Apache 2.0 licence. This is evidence of available code and published guidance, not evidence that the toolkit was executed for this article or that any application passed evaluation.

The starter kit identifies January 2026; its exact publication day is not established here. The structured publication date is left empty rather than inventing a day. The repository is also mutable. Teams reusing it should retain the actual revision, licence and dependency notices associated with their local evaluation.

An original illustrative service answers questions from an employee handbook. The owner first defines acceptable behaviour: supported answers should cite the handbook, missing information should remain explicit and the model should not invent an entitlement. The team prepares de-identified cases representing ordinary questions, ambiguous wording and outdated material. These task-specific expectations give the toolkit something meaningful to test.

Generic benchmarks can complement that set. They may reveal recurring weaknesses or make comparisons easier, but they do not cover the handbook’s actual language or update process. A high benchmark result does not establish that the deployed retrieval service finds the right document. The application’s search, prompt, interface and access controls remain part of the evaluation.

Scoring needs review. An automated judge can help organise results, but it may miss an unsupported qualification or disagree with a human expert. Inspect a sample of passed and failed cases. Record the scoring method and investigate disagreement. If a label is wrong, correct the evaluation process before interpreting its aggregate result.

Repeatability requires configuration discipline. Save the model route, version where available, prompt, dataset and relevant generation settings. Keep credentials outside reports and repositories. Record failures as failures instead of replacing missing outputs with successful-looking placeholders. A comparison between two models is weak if the surrounding conditions quietly changed.

The test environment should be authorised and bounded. Use sandbox data and restrict action permissions. A document-answering assistant can be evaluated without letting it send messages or edit live employee records. When action features are introduced, add tests for authorisation, confirmation, duplicate operations and recovery. The model’s text performance does not establish those application properties.

Costs need visibility. Running many cases can consume provider quota and generate sensitive outputs if the dataset is poorly prepared. Plan the scope and retain enough results to support review without collecting unnecessary private material. A local open-source toolkit may be free to reuse under its licence while model calls, infrastructure and staff review still have costs.

The benefit is a more reproducible basis for choosing and improving an AI service. The failure is a report that looks rigorous because it contains many numbers but lacks relevant cases or trustworthy scoring. Test coverage should be explained in ordinary language, with clear limits. An application owner should be able to identify the important behaviour that remains untested.

Moonshot and the starter kit make evaluation more accessible. The responsible adoption step is to use the actual licensed implementation, preserve its evidence and connect the results to the service’s intended consequences. That turns a toolkit into a useful engineering instrument while avoiding the unsupported claim that running it supplies universal safety approval.

A short review can end with concrete decisions: keep a feature within its current scope, correct a failure, gather missing evidence or stop an unsafe action path. Those decisions matter more than an unexplained overall score, because they show how testing changes the service people will use.

Keep failed cases alongside successful ones so reviewers can see which behaviour improved after a correction and which concerns remain unresolved.