TreasuryQA tests whether AI agents can recover exact figures from scanned financial records

TreasuryQA tests how accurately agents retrieve figures from scanned public records.

TreasuryQA Bench puts agents in front of an archive of 1,435 scanned US Treasury publications and asks them to answer 70 questions with a figure. The October 28 preprint evaluates ten combinations of models and agent software. Each has to find a relevant file through an MCP connection, work with scanned pages and return the requested number. The best observed single-attempt result was 64.8%. [TreasuryQA preprint](https://arxiv.org/abs/2610.38924)

Finding the right document does not finish the job

The benchmark uses a Dropbox replica and gives agents access to document-reading tools. An agent can retrieve the correct volume and still read a faded digit as a different number or take a value from the wrong row. The authors found that most failures returned an incorrect number, with digit errors the most frequent cause. Performance fell on the blurriest scans.

The leading model and harness combination did not have a statistically significant advantage over the second. Each result reflects a bundle of model, tool choices and execution settings. Treating the table as a ranking of models in general would miss the central feature of the benchmark. The workflow tests how each agent uses tools to get from a question to an answer. [TreasuryQA preprint](https://arxiv.org/abs/2610.38924)

What a financial records workflow can learn from the test

A system can report a useful result by giving the figure, the document and the page, while flagging a number it cannot read confidently. A reviewer then has a way to check the row, the unit and any total that should reconcile with the answer. That check is harder when an agent returns only a clean-looking number.

The study’s archive covers Treasury material and tests exact-value retrieval. An institution moving to its own collection would need to account for differences in scan quality, document layout and permissions. The benchmark illustrates how easy it is to overestimate reliability when evaluation starts with pre-extracted text or hands the agent the correct page. [TreasuryQA preprint](https://arxiv.org/abs/2610.38924)