What a controlled banking AI trial should actually measure

Design a narrow experiment around customer outcomes, rollback and evidence rather than a persuasive demo.

A banking AI demonstration can make difficult work look effortless. A well-chosen question gets a fluent answer, the screen moves on, and nobody sees the cases that failed. A controlled trial needs a less theatrical design: a defined task, representative awkward cases and a way to stop without leaving customers stranded.

The FCA describes live AI testing with technical and regulatory support. JPMorganChase reports an internal platform rollout. The UK financial-services survey records both use and third-party dependencies. None is blanket approval for another bank's product or a guarantee of customer outcomes. [FCA testing](https://www.fca.org.uk/publications/feedback-statements/fs25-5-ai-live-testing), [bank deployment](https://www.jpmorganchase.com/about/technology/blog/llmsuite-ab-award), [survey](https://www.bankofengland.co.uk/report/2024/artificial-intelligence-in-uk-financial-services-2024).

Take a fictional bank testing an assistant that drafts responses to routine service questions for staff review. The scope excludes credit decisions, fund movement and changes to identity information. A task description states what the assistant may read, the response it may propose and who approves it. This makes it possible to evaluate the service without pretending that every banking use of AI carries the same consequences.

The bank starts with invented requests written by staff from several roles. A direct address-change question is paired with a cross-border case, a joint-account case and a request missing the account type. Reviewers know the approved answer or acceptable escalation before the trial. They mark unsupported instructions, missing clarifications and unnecessary disclosure separately. One blended accuracy number would conceal the difference between a typo and a serious process error.

The next stage is a shadow exercise using appropriately approved material. The existing process continues to handle requests. The assistant produces drafts in parallel, with no customer-facing authority. Reviewers record what they would change and why. They also note whether finding the error took longer than drafting the response manually. A system can be technically accurate and still add work through opaque explanations.

Design the stop condition before starting. The fictional bank stops the trial if an answer exposes restricted material, proposes an unauthorised action or cannot identify its supporting procedure. This is an example of internal evaluation criteria, not a regulatory threshold. The team keeps the evidence needed to investigate, excludes sensitive content from broad reports and resumes only after the problem is understood and the relevant cases pass.

Test a provider outage deliberately within an authorised environment. Staff should retain their draft and use the normal process. The interface can say that suggestions are temporarily unavailable and provide a safe retry. It should not show a raw provider error or imply that a customer request was completed. If the assistant later returns, it must not silently overwrite a response the staff member finished manually.

A linked workflow introduces new questions. If the assistant can create a service ticket, the trial needs to verify a real receipt from the case system and prevent duplicate tickets on retry. If it can retrieve documents, permissions need separate tests. Success in one capability does not establish the safety of the whole chain. The bank records each boundary so a reviewer can see where generation ends and an actual system action begins.

Measures should include customer-relevant consequences: correct next steps, complete issue capture, safe escalation and avoidable repeat contact. Pair those with staff effort and the proportion of drafts needing material correction. Analyse different case types rather than assuming an overall average applies to everyone. A small trial can show promising behaviour while remaining too limited to support a wider rollout.

The FCA page has dated application windows. As checked for this article, it lists a second-cohort closing date of 24 March 2026 and an evaluation report planned for the first quarter of 2027. Those facts must not be presented as an open application invitation in October. A bank should check the programme's current eligibility and status directly before planning participation.

Public bank descriptions do not supply a transferable price. Budget for review, secure records, operational fallback and the cost of running two processes during the trial. The conclusion may be a narrower launch, more work on the document library or no launch yet. A credible experiment gives decision-makers enough evidence to choose among those outcomes, including a clear account of what remains untested.