UST: Businesses must rethink how AI returns are measured

Australian businesses are entering a new phase of AI adoption, where measurable business outcomes matter more than experimentation.

Companies must move beyond AI pilots and usage-based metrics, with customer outcomes, governance and measurable returns needed to turn investment into production-ready solutions, according to VP & Managing Director ANZ at UST, Kumaran C.R.

Often, businesses are approaching AI by identifying a technology problem and looking for a solution, instead of examining the underlying challenges facing customers and determining where AI could deliver the greatest value.

Some organisations are focused on solving a narrow business problem without considering how the solution would integrate into existing operations or scale beyond an initial pilot.

"The proof of concept itself should be looked at from a business outcome standpoint, from a customer experience standpoint," Kumaran C.R. said.

Kumaran C.R.'s comments reflect the broader challenge facing enterprises as they seek returns from AI investments, with CFOs and CISOs across sectors facing pressure to demonstrate that experimentation can translate into some combination of operational improvements, revenue growth or reduced costs.

Businesses need to reconsider how they select AI use cases, develop pilots and assess whether projects are ready for deployment.

Rather than treating a successful demonstration as the end goal, organisations should establish the business metrics they intend to improve and build solutions around those objectives from the beginning.

AI pilots must deliver measurable returns

UST is encouraging customers to adopt a customer mission-based approach, starting with the outcome customers want to achieve before determining which combination of technology, AI and human involvement could deliver it.

For example, a customer contacting a business should be able to resolve an issue quickly without waiting in a queue, repeating information to multiple representatives or spending unnecessary time navigating different systems.

Businesses should then work backwards from that experience to determine which processes need to change, what technology is appropriate and how success should be measured.

Potential returns include higher customer retention, increased sales, reduced operating costs or improvements in service delivery.

Kumaran C.R. said the approach also required organisations to examine whether their internal systems and processes could support the customer-facing improvements they wanted to introduce.

Pilots need to be tested against realistic operating conditions, including integration requirements, scalability, costs and customer feedback, not simply demonstrating that a particular AI capability was technically possible, helping businesses establish whether a proposed solution can be deployed successfully and deliver ROI.

Kumaran C.R. identified three categories for measuring AI returns: revenue growth and market expansion, cost reduction and operational efficiency, and risk management, regulation and compliance.

The relative importance of each category depends on the organisation, its industry and its maturity.

Energy distributors operating in B2B markets may place greater emphasis on operational efficiency, asset management and capital expenditure than on expanding a conventional consumer customer base.

Businesses operating in a B2C market would prioritise retaining customers, improving renewal rates or reducing the cost of serving existing accounts.

Governance, IP concerns persist

Beyond ROI and cost factors, governance has been identified by UST as another significant obstacle to company-wide AI deployment, particularly as businesses begin introducing AI agents capable of performing increasingly complex tasks with greater autonomy.

Many organisations still operate with technology, data, employees and customer support functions separated into different silos, making it difficult to establish clear responsibility for decisions and actions across the business.

Established governance protocols often depend on employees coordinating these different functions. 

Introducing AI agents only creates additional questions about who is responsible for their actions, how they access information and how their behaviour is monitored.

Kumaran C.R. argued that businesses need stronger visibility into how data is being accessed and used, alongside clear ownership and accountability for AI systems.

Without these controls in place, organisations will very likely struggle to provide the assurances required to move experimental projects into production.

Kumaran C.R. also highlighted intellectual property as a concern for more technologically mature organisations.

Businesses may be reluctant to expose proprietary information, internal processes or commercially valuable data while using external AI platforms, particularly if doing so could weaken their competitive advantage.

Companies should consider where their business value would reside as they increasingly rely on external AI and infrastructure providers.

Cybersecurity adds another layer of complexity, with organisations needing to understand how expanding AI deployments could affect their exposure to threats and ability to respond to incidents.

Cybersecurity resilience is becoming increasingly important alongside prevention, particularly given organisations' regulatory obligations and commitments to customers, Kumaran C.R. argued.

Governance frameworks must also evolve as AI technology changes, rather than being treated as arrangements that can be established once and left unchanged.

UST is therefore advocating for an ongoing framework for governing AI agents, covering their interactions with employees and technology, as well as their access to organisational data.

Such a framework will help provide the confidence businesses need to transition from the AI pilot stage to wider deployment.

Enthusiasm gives way to pressure for value

Kumaran C.R. said the fear of missing out is also contributing to ineffective AI spending, with some organisations measuring progress through technology adoption and consumption rather than business results.

Some companies report the number of AI licences purchased, employees using the technology or even tokens consumed as evidence of progress.

However, those metrics do not necessarily demonstrate that AI is improving business performance.

"Look at how many tokens we're using, look at how much we're doing," Kumaran C.R. said, describing the type of measures some organisations were using to demonstrate AI activity to leadership teams and boards.

The pressure to show that businesses are embracing AI could lead technology teams to prioritise adoption and experimentation without establishing what the investment was expected to achieve.

Consumption-based pricing can often compound the problem, particularly when organisations make AI tools available to large numbers of employees without clearly defined use cases or controls on usage.

The result can be substantial expenditure without a corresponding improvement in productivity, revenue or customer experience.

But Kumaran C.R. explained businesses are beginning to move away from this approach as executives and business leaders demand clearer evidence of value.

"We're seeing a lot of fear of missing out moving into fear of no value, and hence, 'How do we cut back and focus on the right outcome?'" he said.