Matthew Chan | Rainhoffer International | 27 October 2026

Two Australian automation failures, a decade apart, point to five questions that senior leaders should answer – and their boards should be asking – as AI moves into the decisions that matter most.

Matthew Chan is Principal Adviser at Rainhoffer International. This article draws on his experience in financial services since 2007, beginning with the automation and digitisation of post-trade processes at DTCC/Omgeo, and on his public-policy work as well as the evaluation and financial oversight of public-sector programs and orgnisations, and advising financial institutions on AI and technology governance regulations across Asia Pacific. He is a Graduate of the Australian Institute of Company Directors.

In Australia’s home-care system for older people, an algorithm decides how much government funding each person receives. Trained assessors gather the information it uses, but they cannot change the result, even when they judge it clinically inappropriate. The only route to a different outcome is a temporary pathway for departmental review in exceptional circumstances, legislated this month (Commonwealth Ombudsman, “Automation and oversight in Aged Care“, October 2026).

The tool is rules-based automation, not AI, and it sits in government. But the dilemma is the same. The question it raises applies to every organisation automating — whether through AI or not — credit decisions, claims, customer service or operations: when the system is wrong, who can see it, and who can change the outcome? The answer depends less on the technology than on how the organisation is designed around it. Research at UNSW Business School treats an organisation’s ability to absorb disruption, including technological change, as a capacity that has to be built and renewed (Yenni Tim and Dorothy E. Leidner, “Digital Resilience: A Conceptual Framework for Information Systems Research“, Journal of the Association for Information Systems, 2023).

My own work with digitising operational processes, and of firms I have advised or worked within as a consultant, points the same way. Results are often mixed, and the causes mostly organisational: a problem was defined clearly, but against a narrow set of success criteria, while too narrow a group were involved in defining it, and in evaluating the outcome.

In one case, a firm evaluating generative-AI tools for global use Inadvertently framed the exercise around head office needs. Colleagues in out-of -the way markets were included in the problem definition, and in testing — until it was resognised that some of those markets had no access to the tools being tested and, when the issue of access was raised, affected people were bumped off the test group, and the project moved on without them.

Drawing on those cases and on the Australian experience, five questions stand out for leaders evolving their organisations for AI.

1. Has our organisation defined the problem, and who was involved in defining it? How a programme is framed decides which risks get assessed. Robodebt, an automated welfare-debt scheme introduced in 2016 under the former Coalition government, was framed as a savings measure expected to recover more than A$1 billion. It averaged annual tax data into fortnightly income and raised debts on the difference. Whether an average was evidence of a debt at all never became part of the case. The Royal Commission called the scheme “a crude and cruel mechanism, neither fair nor legal” (Royal Commission into the Robodebt Scheme, “Report“, 7 July 2023). It resulted in suicides.

In business the failure is usually less dramatic and more common: a problem statement written by a small group at the centre, reflecting the centre’s view of the work. Practitioners at the launch of UNSW’s PRAxIS Lab made the point that responsible adoption starts with what a technology is needed for, and brings in the people affected from the outset (Barney Tan and Yenni Tim, “The challenges of implementing new technology safely – and fairly“, UNSW BusinessThink, 9 February 2025). For international groups, that means regional offices and other markets. They see regulatory, client and practical constraints that a head-office team may not, including which tools are available to them at all.

2. Do people still have the level of judgement capacity the system assumes as the organisation’s use of technology evolves? Every automated process assumes that someone can tell when it is wrong. That judgement does not transfer automatically from one generation of technology to the next, including AI. In banking, insurance and professional services it is built through years of junior work on exactly the tasks now being automated. A firm can keep its experienced reviewers today while removing the work through which their successors would otherwise have learnt. Few appear to be planning for this. In one recent survey, only 15 per cent of leaders said their organisation was integrating AI strategy with talent strategy, although the sample was small and self-reported (Russell Reynolds Associates and EqualAI, “The AI Governance Leadership Advantage“, 2026).

This is why “human in the loop” can be a weaker safeguard than it sounds. A person can intervene only if they can still recognise when the system is wrong, and AI adoption can erode that ability in two distinct ways.

The first is what is popularly called “brain rot”: the gradual loss of skill and engagement when thinking is routinely handed to a machine. In a small MIT Media Lab study, people who wrote essays with an AI assistant showed weaker brain connectivity than those who worked unaided, and had difficulty recalling what they had written (Kosmyna and colleagues, MIT Media Lab, 10 June 2025). The second is “brain fry”: acute mental fatigue from using or overseeing AI tools beyond a person’s cognitive capacity. In a survey of 1,488 US workers, those overseeing several AI agents reported the greatest strain, and those who felt they used too much AI reported more errors (Bedard, Kellerman and colleagues, Harvard Business Review, March 2026). The first erodes judgement over time; the second impairs it on the day. Either way, the person in the loop still signs off but can no longer genuinely intervene. Leaders need to manage both, by keeping people practising the judgement they are expected to exercise and by sizing oversight workloads to what people can realistically review.

3. Can the people closest to the decision intervene and act on what they see? This is where the aged-care system failed. It had the technology, 22 assessment providers and trained assessors, and the tool had even been built with an override. Shortly before the new Aged Care Act took effect on 1 November 2025, the department concluded that the rules gave assessors no legal discretion to use it (The Weekly SOURCE, “Freedom of information documents reveal how human override disappeared from Support at Home assessments“, 15 September 2026). Depending on the period, between a quarter and more than two-fifths of decisions taken to internal review were then set aside or varied, while only about 0.6 per cent of assessments were audited. The department disputes several of the Ombudsman’s findings, and the government says it is already acting on many of the recommendations (The Weekly SOURCE, “Ombudsman and Health Dept Secretary clash over IAT review“, 26 October 2026).

The corporate version is familiar. A bank keeps experienced credit officers alongside a decisioning model, but makes overriding the score so costly to document that it rarely happens. An insurer keeps claims handlers in an automated triage process, but measures them on throughput. Or, culturally, staff made to feel like they are not embracing AI if they questioning its output, or are measured by blunt KPI such as I have seen in a number of global banks, where the main KPI is hours spent using AI tools. In such cases a person remains in the loop, yet the conditions for their judgement to change the outcome have gone. More technology or more training will not fix this. Culture, leadership and clear decision rights will.

4. Is oversight built into the work, or added at the end? Oversight is often treated as a brake on delivery. Over any meaningful horizon the reverse holds. Without oversight, trust in a system falls; without trust, adoption stalls; and without adoption, the promised efficiency does not arrive. Retrofitting is expensive. The aged-care pathway passed by Parliament on 15 October 2026 applies only in “exceptional circumstances” and only to decisions made on or before 31 December 2027 (The Weekly SOURCE, “First Nations Aged Care Commissioner and IAT escalation pathway become law with expiry date“, 16 October 2026). The Ombudsman is still recommending legislation to let assessors substitute their own decisions.

For firms now deploying AI agents, the default design, in which an agent completes the task and a person signs it off, is the one most likely to make oversight nominal. The reviewer sees only the output, at volume and under time pressure. Placing people at defined points in the workflow, where their knowledge of context matters most, makes oversight real and shows management where the system can be trusted.

5. Will we know when it is going wrong? Most AI governance effort goes into committees, policies and approval gates. These matter, but both Australian automation cases failed further out, in the feedback loops and incentives that determine whether warnings travelled and were heeded. In the Robodebt case, the Department of Social Services was advised in November 2014 that income averaging did not accord with the legislation, and the Administrative Appeals Tribunal found the method unlawful in March 2017. The scheme continued. A decade later, the Ombudsman found the aged-care system inconsistent with the Royal Commission’s recommendations on transparency in automated decision-making. In both cases principles had been written down, but they had not been turned into decision rules, review points, escalation routes and feedback loops.

Boards have guidance on what to look for. The Australian Institute of Company Directors includes people, skills and culture, and monitoring, reporting and evaluation, among its eight elements of AI governance (Australian Institute of Company Directors and Human Technology Institute, “A Director’s Guide to AI Governance“, 12 June 2024).

Regulators are pressing the same points. In the financial services context, APRA has found that some institutions depend heavily on single vendors whose embedded AI reduces “transparency over where and how models are trained, updated or constrained”, and that many boards lack the technical literacy to challenge management (Australian Prudential Regulation Authority, “APRA calls for a step-change in AI-related risk management and governance“, 30 April 2026). From 10 December 2026, organisations covered by Australia’s Privacy Act must also disclose the kinds of significant decisions that computer programs make or support (Office of the Australian Information Commissioner, “New resources on transparency for use of AI and automated decision-making“, 30 September 2026).

None of this argues for slowing down. Organisations that can answer these five questions can automate and scale AI with more confidence and less rework. A practical first step is to take one consequential AI initiative, test it against all five, and put effort into the weakest answer rather than defaulting to more technology or more training.

The final test is simple: can management show how a wrong outcome would be detected and corrected, and by whom? In neither Robodebt nor the aged-care case did the software malfunction. What was missing was an organisation and the leadership able to recognise and correct outcomes that were wrong for the people affected.