The productivity gains delivered by AI are being eroded because finance professionals often reject outputs they cannot explain or validate. We look at how the ‘trust gap’ in existing AI models can be addressed by designing more transparent, user-focused systems.
Artificial intelligence (AI) has the potential to deliver significant improvements to efficiency and productivity across a wide range of business activities – including accountancy and finance. But
new data shows that the adoption of generative and agentic AI services by finance professionals is being slowed by concerns related to trust, transparency and reliability.
Research carried out by Sage, in partnership with market-intelligence firm IDC, found almost three-quarters of finance leaders (71%) would reject an AI system if it was unable to fully explain how it arrived at its outputs. The average finance professional spends almost 13 hours a week reconstructing, validating and defending AI outputs, the study found. As a direct result, 26% of the time savings created by AI are lost.
These findings reveal how AI take-up is being impeded by systems that are often run on a ‘black box’ principle, characterised by a lack of transparency around how the model makes calculations and arrives at judgements.
Lee Doughty, director of security, resilience and AI practice at technology consultancy
Intercity, says the fact that AI outputs often appear confident and persuasive, even when they are incorrect, is a major source of problems.
“In areas such as finance, reporting and compliance, this creates understandable caution and concern, because a believable error can quickly become a real business issue,” he warns.
“From what we see, the underlying issues are consistent, with many organisations lacking visibility of how AI is being used and on what data. Governance and security controls are often applied after the fact rather than by design and this must change.
AI is introduced into workflows without clear guidance on validation, and this leads into little connection between AI activity and measurable business outcomes.”
Seb Kirk, CEO and co-founder of AI solutions platform
GaiaLens, adds:
“Businesses are understandably cautious about AI outputs, particularly in finance and other high-stakes functions where accuracy, accountability and auditability matter. The core issue is that many AI systems generate answers without clearly showing how conclusions were reached, what data was used or whether the underlying information is reliable.”
In such cases, Kirk says, workers need to spend additional time manually checking and verifying outputs. “Trust in AI comes less from the sophistication of the model and more from the quality of governance surrounding it. Organisations need explainable systems, clear data lineage and traceable audit trails so users can understand how recommendations are produced.”
Andrew Graham, business development manager at information-management company
PFU says that, for SMBs in particular, the focus should be not just on adding AI tools to their processes, but on improving data quality and workflow integrity first.
“AI systems are only as reliable as the information they receive,” he points out.
The time and effort being devoted to validating AI outputs can be described as the ‘trust cost’ of AI –the gap between what AI systems promise to deliver, and what finance teams can rely on in practice.
The challenges that finance professionals are facing in maximising the benefits of AI have led Sage to make a fresh commitment to tackling this AI trust gap: the company has just announced a new partnership with PwC that will redefine how AI solutions for finance and accountancy are designed and implemented.
This initiative, ‘Beyond the Black Box’, was announced at April’s
Sage Future event in San Francisco and has been created with small and medium-sized businesses in mind. It aims to create ‘glass box’ AI systems, where every answer is explainable and every output can be quickly and transparently verified.
"Finance does not run on answers alone – it runs on answers you can explain," says Steve Hare, CEO of Sage.
"If you cannot show how a number was produced, you cannot use it. That is why we are building AI differently. AI you can trust can’t be a black box: we see it as a glass box that gives finance teams full visibility into how it works, so they can stand behind it with confidence.”
AI is set to have a transformational impact on how finance professionals operate, increasing efficiency and freeing up time to devote to higher-value work. But in order to unlock AI’s true potential, it is vital that finance leaders are able to fully understand and trust the decisions and outputs that AI services make.
