Australian firms race ahead on AI despite oversight gaps

Australian organisations are pushing AI applications into production despite major gaps in oversight, according to new research from Ecosystm commissioned by Avocado and Dynatrace. The study found that 91% of surveyed organisations have AI applications in development, pilot or live use.

It also suggests that most cannot fully explain how their AI systems behave in production or who is responsible when something goes wrong. Only 9% of respondents said they had end-to-end visibility into the behaviour of AI applications in live environments.

The findings are based on a survey of 154 technology leaders and engineering practitioners at Australian organisations with more than 500 employees. The report focuses only on Australian data, rather than broader regional averages.

That points to a wide gap between board-level approval of AI initiatives and the operational controls needed to account for failures, security breaches and unexpected outputs. For directors and executives, the issue is less whether AI is being adopted than whether the organisation can explain decisions, trace data sources and respond quickly when problems emerge.

Governance gap

Security and guardrail violations were among the top behaviours respondents said they monitor, cited by 47%. Retrieval performance ranked first. But the report indicates that monitoring warning signs is not the same as having enough visibility to investigate and resolve incidents end to end.

"We're seeing Australian boards ask good questions about AI strategy and ROI, and far fewer asking whether their organisation could actually explain an AI failure if a regulator or customer demanded an answer tomorrow. That's the question this data should put on the agenda," said Zana Stojanovski, GM Business Solutions, Avocado.

The research also found that operational maturity drops as organisations move from spotting a problem to understanding and fixing it. A quarter of respondents said they could detect AI-related issues quickly or proactively. That fell to 23% for diagnosing root causes in near real time and 17% for resolving issues quickly or through automation.

In practical terms, many organisations may notice that an AI service is malfunctioning, but far fewer can explain why within a meaningful timeframe. Fewer still can contain or fix the issue before regulatory, financial or reputational damage spreads.

"Detection without fast diagnosis and resolution isn't real accountability; it's just an early warning that a problem is already underway. Boards should be asking not 'can we tell something's wrong,' but 'how long between something going wrong and us being able to explain and fix it' because that's the window where cost, damage and reputational risk actually accumulate," said Stojanovski.

Fragmented ownership

The study points to another weakness in how large organisations manage AI systems: responsibility is often split across several teams. Site reliability engineers handle monitoring in some businesses, while MLOps, AI engineering and data science teams take on that role elsewhere.

What remains uncommon is a single, joined-up model of accountability covering model behaviour, data quality, application logic, infrastructure and security. That fragmentation creates a governance problem when boards, regulators or customers want a clear answer on who owns the risk.

The report argues that ownership matters as much as technical monitoring. Better tooling and processes can improve explainability and response speed, but responsibility still has to be explicitly assigned.

Observability limits

Most large Australian organisations already collect substantial production telemetry, according to the research, but only 27% said that information had significantly changed how software is designed, built and tested from the outset. For many, telemetry remains a troubleshooting tool after deployment rather than something that guides earlier decisions.

Developers appear to feel that shortfall directly. More than half, 53%, said easier access to logs, traces and runtime data during debugging would improve their daily experience.

"Observability needs to move upstream. Engineering teams need production context when they are designing, building and testing software, not after an issue reaches production. As AI increases the complexity of development and operations, that visibility becomes an engineering capability as well as a governance requirement," said Darian Bird, Principal Advisor, Ecosystm.

The report says traditional monitoring tools were not designed for AI-specific problems such as model drift, hallucinated responses or hidden cost spikes within ordinary traffic. Without more tailored visibility, even formal governance structures may struggle to answer basic questions after an incident.

That leaves organisations in a bind. Governance without supporting evidence offers little help when scrutiny arrives, while data without a clear owner can leave warning signs unactioned.

"Most organisations think they've solved observability once the dashboards exist. The real test is whether that data changes a decision before something breaks - in design, in code review, in testing - not just after an incident, when it's too late to prevent the cost. That's the gap between having telemetry and actually operating on it," said Stojanovski.