AI & TECHNOLOGY
Before scaling finance AI, build a test it can fail
A convincing demonstration shows what an assistant can do. A useful evaluation shows when to trust the workflow, when to review it and when to stop it.
By CA Tony Thomas · Chartered Accountant · 4 minute read
Prepared
PRACTICAL PERSPECTIVES / FINANCE · AUTOMATION · AI
Define the task before testing the model
A finance AI demonstration can look impressive when the document is clean, the question is straightforward and the expected answer is already known. Production work introduces missing attachments, conflicting periods, unfamiliar wording and questions the system should decline to answer.
Evaluate a narrow task first. “Summarise payment terms from approved contracts” is more testable than “help with finance.” Specify the inputs, permitted output, evidence requirements, reviewer and actions the assistant cannot take.
NIST's Generative AI Profile identifies risks including confidently incorrect content. The practical implication for this proposed workflow is to test factual support and behaviour under uncertainty, rather than relying on fluent language or a reassuring confidence score.
Create a representative case set
For a fictional contract-summary assistant, assemble an authorised evaluation set containing clean contracts, amendments, scanned documents, missing schedules and contradictory terms. Use synthetic records where possible and keep real documents within approved controls.
Have qualified reviewers establish the expected answer and supporting passages before running the assistant. Some cases should have no definitive answer. The expected result for a contract with a missing payment schedule may be “insufficient evidence,” together with a request for that schedule.
Separate development cases from a held-out evaluation set. Repeatedly adjusting the prompt against the same examples can make the workflow appear stronger than it is. Keep a record of what each version has already seen.
Include ordinary cases in proportion to likely use, but add a dedicated challenge set for low-frequency, high-impact failures. Report those results separately so common easy cases do not hide a critical weakness.
Score the claim, the source and the decision
Suppose the test contains 40 cases: 30 answerable and ten intentionally incomplete. If the assistant answers 28 of the answerable cases correctly and fabricates a term in four incomplete cases, reporting “28 correct answers” says little about deployment readiness.
Measure correct supported answers on answerable cases, correct abstention on incomplete cases, and the severity of unsupported claims. A quoted payment period needs the right document, section, entity and effective version. An authentic citation attached to the wrong clause is still a failure.
Distinguish extraction errors from interpretation errors. Misreading “60” as “30” needs a different response from selecting a superseded amendment. Both can affect the decision, but one may require better document processing while the other requires stronger version controls.
A single severe failure can justify stopping a release even when the overall average is high. Define the release criteria with the accountable control owner before seeing the results.
Test behaviour beyond normal questions
Add cases in which a document contains instructions telling the assistant to ignore policy or reveal other records. The document is evidence, not authority to change the assistant's permissions. Check that outputs respect the user's access rights and do not disclose unrelated information.
Test repeated and paraphrased questions, long documents, unfamiliar layouts and ambiguous dates. Ask for information that is absent. Verify that the system can distinguish a missing fact from a fact retrieved from another contract.
Run the same case more than once where variable outputs are possible, and retain the observed range of behaviour. An occasional severe error should not be removed from the record because the next attempt happened to be correct.
Assess the complete workflow, including retrieval, document extraction, model response and human review. Testing the language model alone does not establish that the end-to-end process is safe.
Measure the reviewer, too
Give reviewers a structured output with the source passage, uncertainty and required decision. Measure total review time and the number of corrections, not just response-generation speed.
Where feasible, compare AI-assisted work with the established process using similar cases and independently reviewed outcomes. Do not attribute every improvement to the model when the pilot also introduced cleaner templates or better source data.
Track cases where reviewers accepted a plausible but wrong answer. Human review is a control only when people have the evidence, competence and time to challenge the result. A button labelled “approve” is not sufficient.
Make release conditional and reversible
Document the approved task, evaluation results, unresolved limitations, model and prompt versions, data sources and responsible owner. Start with restricted access and a manual fallback. Prevent the assistant from creating journals, payments or external communications unless separately authorised and controlled.
Re-evaluate when the model, retrieval rules, documents or task changes materially. Keep incident and correction records so production failures become new test cases rather than forgotten anecdotes.
A strong evaluation does not prove that the system will never fail. It creates a defensible basis for a bounded use, makes the known limits visible and gives the organisation a clear reason to pause when those limits are exceeded.
Sources & further reading
- NIST: Generative AI Profile
Voluntary cross-sector guidance on generative AI risks. The contract evaluation design above is an original illustrative proposal.
- AICPA: AI considerations for financial reporting and auditing
The October 2026 announcement provides professional context on AI-related risks and controls; it does not certify the proposed workflow.
Linked materials checked on 1 November 2026. They provide background; the worked examples and workflow designs are original.
Personal educational perspective, independent of my employer. All business examples and figures are fictional and illustrative. Proposed workflows do not claim completed deployments or measured productivity gains. This is not entity-specific accounting, tax, legal or investment advice. Apply professional judgment, the applicable reporting framework and your organisation’s approval and data-protection policies.